There are many excellent books and articles describing those topics and how they can be implemented in various software frameworks, and those descriptions will not be repeated here. There also are many articles on Big Tech implementing AI at scale. But how do "regular" organizations implement AI projects successfully, especially within an existing portfolio of solutions? In the BLOG@CACM post "Anna Karenina on Development Methodologies," I described how the famous opening line "happy families are all alike, unhappy families are unhappy each in their own way" applies to software development. This post will describe in a similar vein the development behaviors with the highest chance of success for AI efforts.

As we've seen in our previous posts, developing software is not only about writing code. You have to use quality tools to write human and machine readable and maintainable code and you have to automate your workflow while following industry best practices (i.e. using git, doing unit tests, and etc.) We can't emphasize enough that software development is team work, and there are development methodologies which help you to manage the complexity of a project and also these methodologies help you to coordinate you and your co-workers' effort to make a good software. Following various software development methodologies is very similar to ideological fundamentalism. As Heraclitus said, you cannot step into the same river twice. Also, you cannot do the same project twice, so you cannot face the challenges coming up during a project twice.

View Vinayak Chaturvedi’s profile on LinkedIn, the world’s largest professional community. Vinayak has 2 jobs listed on their profile. See the complete profile on LinkedIn and discover Vinayak’s connections and jobs at similar companies.

Fullstack Developer Fullstack Developer (State Street Bank and Trust Company; Boston, Massachusetts): will be a senior developer who works on the State Street Risk and Regulatory Technology Applications Development team, a team that is building next generation risk management capabilities to quantify, aggregate, and report risk measures. The Fullstack Developer will develop applications that synthesize data from State Street's varied trading and reference systems, perform simulation, pricing, and aggregation, and report the results for regulatory and risk management purposes. The Fullstack Developer will lead a team of geographically distributed developers to deliver solutions to State Street business problems in the risk domain, and is expected be on call to be an escalation contact if there are production failures. Specific duties of the position include: Designing and implementing fault tolerant architectures to optimize operational resiliency for both batch and real-time environments; performing component design using object-oriented paradigms; performing hands-on proof-of-concept development with Big-Data technologies, specifically in the areas of statistical processing of large volume numeric data, in-memory aggregation, and machine learning; understanding the structure of existing systems to accomplish tactical and strategic business objectives within aggressive timelines; developing scrum master skills/Agile development methodology in engineering teams; mentoring junior developers across geographically diverse locations and identifying and addressing issues and conflicts in an effective way; managing a matrix of geographically distributed developers and architecting large scale systems; designing and implementing High Volume data processing stack including end to end processing structures for data ingestion, transformation, statistical analytics and reporting/UI; working with advanced database programming including building generic frameworks, Oracle queues, and object collections. Minimum requirements are: Bachelor's degree, or its equivalent, in Computer Science, Computer Engineering, or another directly related technical field; and 5 years of experience in software engineering.

DevOps and agile programming continue to make inroads into software-development teams, with the two development methodologies accounting for more than two-thirds (68%) of the practices at companies polled in a recent survey, according to a report published by development-tools maker GitLab on Tuesday. The adoption coincides with developers taking an increasing role in securing software -- so-called "shifting left" -- with 39% of developers "feeling fully response for security," up from 28% last year, while 32% share responsibility for security with other teams, according to survey results. Overall, the security outlook among developers has increased significantly over the past year, with 72% calling their organization's security either "good" or "strong," up from 59% the prior year. This year, more than any other year, integrating security into DevOps -- often called DevSecOps, SecDevOps, or secure DevOps -- is a reality, says Johnathan Hunt, vice president of security at GitLab. "Last year, often no one knew who owned security, and the adoption of DevSecOps was stagnant -- you could see that," he says.

The rapid rate of change enabled by software make this industry more vulnerable than most to the falling behind on the innovation curve. This problem has only accelerated in recent years as the number of disruptive technologies have grown at an exponential rate fueled by the growing size of the market and the number of software engineers. The open source community has been a driving source of disruptive technologies such as big data Hadoop and Spark, JavaScript frameworks like Angular and React, and machine learning frameworks like TensorFlow. Software based companies who do not embrace these disruptive technologies face the ever-increasing risk of being pushed aside by those that do. To make this even more challenging, the skills required to enhance the current product and the skills required to innovate using new disruptive technologies are different.

Data science development is very different from software development, and getting the two to mesh is sometimes like trying to cobble together Tinker Toys with Lincoln Logs. Software development is "Measure twice; cut once," while Data Science is "Cut, cut, cut!" The methodologies and processes that support successful software development do not work for data science projects according to one simple observation: software development knows, with 100% assurance, the expected outcomes, while data science – through data exploration and hypothesis testing, failing and learning – discoversthose outcomes. First introduced in the blog "What's The Difference Between BI Analyst and Data Scientist?", the Data Science Engagement methodology in Figure 1 supports the rapid exploration, rapid testing, and continuous learning Data Science "Scientific Method[1]". Let's review each of these in more detail.

Automation platforms like Pega have capabilities that span tactical intent and strategic intent, namely Pega robotics, cognitive decision making, business function transformation, and analytics. If you include the possible machine learning capabilities residing on top of such platforms, practical adoption patterns and methods vary widely. Traditional development methodologies normally called for, like Agile, need to be used with the understanding that you're customizing your approach for automation initiatives. For example, the most successful preferred method for scaled business transformation would be Distributed Agile, which is nothing but user story-based incremental development done with geographically distributed teams. But if you have Pega robotics as the entry point, the development lifecycle actions follows the sequence of Assess, Record, Design, Unit Test, Verify, Soft Launch, Iterate and Manage Exceptions, and Full Launch.

The practice of software development has created development methodologies such agile development and lean methodology to tackle the complexity of development with the objective of improving the quality and efficiency of software creation. Although Deep Learning is built from software it is a different kind of software and therefore a different kind of methodology is needed. Deep Learning differs most from traditional software development in that a substantial portion of the process involves the machine learning how to achieve objectives. The developer is not completely out of the equation, but is working in concert to tweak the Deep Learning algorithm. Deep Learning is sufficiently rich and complex a subject that a process model or methodology is required to guide a developer.

The development of artificial intelligence systems has to date been largely one of manual labor. This Constructionist approach to A.I. has resulted in a diverse set of isolated solutions to relatively small problems. Small success stories of putting these pieces together in robotics, for example, has made people optimistic that continuing on this path would lead to artificial general intelligence. This is unlikely. "The A.I. problem" has been divided up without much guidance from science or theory, resulting in a fragmentation of the research community and a set of grossly incompatible approaches. Standard software development methods come with serious limitations in scaling; in A.I. the Constructionist approach results in systems with limited domain application and severe performance brittleness. Genuine integration, as required for general intelligence, is therefore practically and theoretically precluded. Yet going beyond current A.I. systems requires significantly more complex integration than attempted to date, especially regarding transversal functions such as attention and learning. The only way to address the challenge is replacing top-down architectural design as a major development methodology with methods focusing on self-generated code and self-organizing architectures. I call this Constructivist A.I., in reference to the self-constructive principles on which it must be based. Methodologies employed for Constructivist A.I. will be very different from today's software development methods. In this paper I describe the argument in detail and examine some of the implications of this impending paradigm shift.