We can work together in data with MLFlow
In the last post about mlflow we see just a little description about mlops and the module of mlflow tracking. In this post i’m gonna write about how you can work in a centralized way in your organization.
In the work environment you always need to share your work with your coworkers, if you need some help, share some knowledge or even a presentation for the c-suite. As you read in the last post, mlflow brings help with the ML lifecycle in your organization, and gives us a possibility to manage the chaos when we work with ML or AI.
So, if you are introducing yourself in the mlops world, this post is probably gonna be very useful. In this video we can get the idea of how to configure the cloud server (vm) to work with this, and I’m gonna put some instructions to configure the remote server.
What i’m talking about
As the mlflow website describes, MLflow runs can be recorded to local files, to a SQLAlchemy compatible database, or remotely to a tracking server. MLflow artifacts can be persisted to local files and a variety of remote file storage solutions. For storing runs and artifacts, MLflow uses two components for storage: backend store and artifact store. While the backend store persists MLflow entities (runs, parameters, metrics, tags, notes, metadata, etc), the artifact store persists artifacts (files, models, images, in-memory objects, or model summary, etc).
So, in the video, present how we can build a vm and work together in different environments and get lots of metrics so we can compare and improve our models in a centralized way. This differs a little from github, because in the data world, not only you work with code: you have a lot of metadata, hyperparameters and more
I work a little in the security layer, just enough to have a user and password and some kind of login. I don’t recommended to put this in production or even dev, but only at your own risk, but if you do this, at least configure your ip to only access to the vm.
You can use your shell o the google cloud shell to run this commands
gcloud compute instances create mlflow-log-server \ --machine-type n2-standard-2 \ --zone us-central1-a \ --tags mlflow-log-server \ --metadata startup-script='#! /bin/bash sudo apt update sudo apt-get -y install tmux echo Installing python3-pip sudo apt install -y python3-pip export PATH="$HOME/.local/bin:$PATH" pip3 install mlflow google-cloud-storage'
if you check, the metadata arg help us to install all the components necessary for mlflow server runs
gcloud compute firewall-rules create mlflow-log-server \ --direction=INGRESS --priority=999 --network=default \ --action=ALLOW --rules=tcp:5000 --source-ranges=0.0.0.0/0 \ --target-tags=mlflow-server
With this you can create a firewall rule.
This happens in the vm
mlflow --version
mlflow server \ --backend-store-uri sqlite:///mlflow.db \ --default-artifact-root gs://<bucket> \ --host <ip-internal>
# install nginx in your vm
sudo apt-get install nginx apache2-utils
# create users
sudo htpasswd -c /etc/nginx/.htpasswd root # you can change sudo htpasswd /etc/nginx/.htpasswd user # you can change
# open a nano editor and modify the nginx configuration
sudo nano /etc/nginx/sites-enabled/default
#just add the lines in the location section
server {
location / {
proxy_pass http://localhost:5000;
auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/.htpasswd;
}
}
#control x to exit
sudo service nginx restart
mlflow server \ --backend-store-uri sqlite:///mlflow.db \ --default-artifact-root gs://<bucket> \ --host localhost
With this you can connect with the <remote-ip> of your vm and check the experimentations
All the work you can do is the same as in the last post, the modification of the script is in the github link.
Thanks for reading, this post is for a series of posts about mlflow.