dependency


Considering How Machine Learning APIs Might Violate Privacy and Security - DZone Security

#artificialintelligence

I was reading about how Carbon Black, an endpoint detection and response (EDR) service, was exposing customer data via a 3rd party API service they were using. The endpoint detection and response provider allows customers to optionally scan system and program files using the VirusTotal service. Make sure we are thinking deeply about what data and content we are making available to platforms via artificial intelligence and machine learning APIs. Make sure we are thinking deeply about what data and content sets we are running through the machine learning APIs and reducing any unnecessary exposure of personal data, content, and media.


The Calculus of Service Availability

Communications of the ACM

As detailed in Site Reliability Engineering: How Google Runs Production Systems1 (hereafter referred to as the SRE book), Google products and services seek high-velocity feature development while maintaining aggressive service-level objectives (SLOs) for availability and responsiveness. Internally at Google, we use the following rule of thumb: critical dependencies must offer one additional 9 relative to your service--in the example case, 99.999% availability--because any service will have several critical dependencies, as well as its own idiosyncratic problems. In such a model, as shown in Figure 1, there are 10 unique first-order dependencies, 100 unique second-order dependencies, 1,000 unique third-order dependencies, and so on, leading to a total of 1,111 unique services even if the architecture is limited to four layers. An error budget is simply 1 minus a service's SLO, so the previously discussed 99.99% available service has a 0.01% "budget" for unavailability.


Data Version Control in Analytics DevOps Paradigm

@machinelearnbot

It makes your data science projects reproducible by automatically building data dependency graph (DAG). Machine Learning modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists' collaboration takes a serious amount of the team's effort. By any mean, DVC is going to be a useful instrument to fill the multiple gaps between the classical in-lab old-school data science practices and growing demands of business to build solid DevOps processes and workflows to streamline mature and persistent data analytics.


R code and reproducible model development with DVC

@machinelearnbot

DVC is an open source tool that could help with achieving code simplicity, readability and faster model development.The idea is to track files/data dependencies during model development in order to facilitate reproducibility and track data files versioning. DVC tool improves and accelerates iterative development and helps to keep track of ML processes and file dependencies in the simple form. On the R example we saw how DVC memorizes dependency graph and based on that graph re executes only jobs that are related to the latest changes. It can also work in multiuser environment where dependency graphs, codes and data can be shared among multiple users.


Data Version Control in Analytics DevOps Paradigm

@machinelearnbot

It makes your data science projects reproducible by automatically building data dependency graph (DAG). Machine Learning modeling is an iterative process and it is extremely important to keep track of your steps, dependencies between the steps, dependencies between your code and data files and all code running arguments. This becomes even more important and complicated in a team environment where data scientists' collaboration takes a serious amount of the team's effort. By any mean, DVC is going to be a useful instrument to fill the multiple gaps between the classical in-lab old-school data science practices and growing demands of business to build solid DevOps processes and workflows to streamline mature and persistent data analytics.


vahidk/EffectiveTensorflow

#artificialintelligence

To get the dynamic shape of the tensor you can call tf.shape op, which returns a tensor representing the shape of the given tensor: The static shape of a tensor can be set with Tensor.set_shape() Use this function only if you know what you are doing, in practice it's safer to do dynamic reshaping with tf.reshape() op: If you feed'a' with values that don't match the shape, you will get an InvalidArgumentError indicating that the number of values fed doesn't match the expected shape. So it's valid to add a tensor of shape [3, 2] to a tensor of shape [3, 1] Broadcasting allows us to perform implicit tiling which makes the code shorter, and more memory efficient, since we don't need to store the result of the tiling operation. In order to concatenate features of varying length we commonly tile the input tensors, concatenate the result and apply some nonlinearity. It allows building dynamic loops in Tensorflow that operate on sequences of variable length.


vahidk/EffectiveTensorflow

@machinelearnbot

To get the dynamic shape of the tensor you can call tf.shape op, which returns a tensor representing the shape of the given tensor: The static shape of a tensor can be set with Tensor.set_shape() Use this function only if you know what you are doing, in practice it's safer to do dynamic reshaping with tf.reshape() op: If you feed'a' with values that don't match the shape, you will get an InvalidArgumentError indicating that the number of values fed doesn't match the expected shape. So it's valid to add a tensor of shape [3, 2] to a tensor of shape [3, 1] Broadcasting allows us to perform implicit tiling which makes the code shorter, and more memory efficient, since we don't need to store the result of the tiling operation. In order to concatenate features of varying length we commonly tile the input tensors, concatenate the result and apply some nonlinearity. It allows building dynamic loops in Tensorflow that operate on sequences of variable length.


How to get started with machine learning: Use TensorFlow

#artificialintelligence

TensorFlow "has made it possible for people without advanced mathematical training to build complex -- and sometimes useful -- models." Not surprisingly, software engineering teams are generally not well-equipped to handle these complexities and so can fail pretty seriously. "That capability, more than anything else -- including deep learning -- has made it possible for people without advanced mathematical training to build complex -- and sometimes useful -- models." This willingness to get hands dirty with open source code is his fourth point, that "successfully deploying machine learning will require that a team is willing to look deeply into how things work."


How to get started with machine learning: Use TensorFlow

#artificialintelligence

TensorFlow "has made it possible for people without advanced mathematical training to build complex -- and sometimes useful -- models." Not surprisingly, software engineering teams are generally not well-equipped to handle these complexities and so can fail pretty seriously. "That capability, more than anything else -- including deep learning -- has made it possible for people without advanced mathematical training to build complex -- and sometimes useful -- models." This willingness to get hands dirty with open source code is his fourth point, that "successfully deploying machine learning will require that a team is willing to look deeply into how things work."


Building--and scaling--a reliable distributed architecture

#artificialintelligence

I recently asked Joseph Breuer and Robert Reta, both Senior Software Engineers at Netflix, to discuss what they have learned through implementing a service at scale at Netflix. Joseph and Robert will be presenting a session on Event Sourcing at Global Scale at Netflix at O'Reilly Velocity Conference, taking place October 1-4 in New York. The primary challenge when operating a service in a distributed architecture at scale is managing for the behavior of your downstream dependencies. Continue reading Building--and scaling--a reliable distributed architecture.