MLOps is a set of best practices that revolve around making machine learning in production more seamless
This picture was taken from state-of-mlops
credits to service https://tableconvert.com/ for formatting the table below
| Comparison Item | MLFlow | AOA | ClearML |
|---|---|---|---|
| Experiment tracking | |||
| Track experiments with api calls | yes | yes | yes |
| Store experiment in metadata | yes | yes | yes |
| Metadaba backend | mysql, mssql, sqlite, and postgresql | h2, mysql, postgresql, mssql | mongo |
| Experiment tracking UI | yes | yes | yes |
| Experiment annotation | yes | only tags and comment on approval | yes |
| Experiment comparison | yes | yes | yes |
| Save graphics with experiments | yes | yes | yes |
| Passing parameters into ml code (training/evaluation) | yes | yes | yes |
| Enironments support | yes, conda and docker | yes, docker | yes, docker, pip, conda, poetry |
| Code repo | |||
| Package ml code into projects with fixed file/folder structure | yes | yes | yes, but the structure or repo is not fixed |
| Clone code from git | yes | yes | yes |
| Monitor changes in ml repo | no | yes | yes |
| Artifact storage | |||
| Storage types supported | S3-compatible, Azure Blob Storage, Google Cloud Storage, FTP server, SFTP Server, NFS, HDFS |
Jfrog Artifactory, S3-compatible | S3-compatible, azure, google cloud, minio |
| Proxy access to artifacts in UI | yes | yes | yes |
| Dataset support | |||
| Supports dataset definitions | no | yes | yes |
| Connection types | n/a | Vantage, S3, Hadoop and a custom type of connections | any |
| Credentials support | n/a | Username/password, Kerberos, AWS Secrets and Kubernetes as credentials | yes |
| Dataset versions | n/a | yes | yes |
| Dataset templates | n/a | yes | no |
| Model Lifecycle | |||
| Support model registry | yes | yes | yes |
| Available model statuses | staging, production, archive | Trained, Evaluated, Deployed, Retired, Approved, Rejected | yes |
| Lineage support | stated in doc but not clear how | yes | yes |
| Integration with BPM | no | yes, JBPM support | no |
| Trigger model events on status change, code change in repo | no | yes | no |
| Model serving | |||
| Support model serving | yes | yes | yes |
| Default model serving engine | conda env, gunicorn | docker, gunicorn | docker, gunicorn |
| Batch scoring support | yes | yes | yes |
| AirFlow integration for batch scoring | no | yes | no |
| Cloud deployment | yes: AzureML, Sagemaker | yes: Teradata Vantage and any other cloud | yes: k8s |
| Bring your own model (BYOM - deploy with no code via onnx, pmml, etc ) | out of the box - no, need customization | yes | |
| Deploy to spark | yes | yes | yes |
| Deploy to K8S | yes, over Azure | yes | yes |
| Deploy to other clusters | yes, with custom plugins | yes, with custom Jenkins workflow | yes, with custom Jenkins workflow |
| Canary deployment | no | no | yes |
| Model monitoring | |||
| Support model monitoring | no, need 3rd party solution like evidently.ai | yes | yes |
| Feature drift | n/a | yes | yes |
| Prediction drift | n/a | yes | yes |
| Model performance | n/a | yes | yes |
| Alerting | n/a | yes, with Prometheus | yes |
| Extra items | |||
| Extensibility with plugins | yes, can write plugins for: Tracking Store ArtifactRepository Run context provider Model Registry Store MLFlow Project backend |
yes with custom Jenkins pipelines | n/a |
| Model hierarchy | no | yes, can pack models into projects | yes |
| Authentication | yes, only with databricks | yes, LDAP | yes |
| Access control | yes, only with databricks | yes | yes |
| Pipeline mechanism | no | no | yes |
| Automatically capture input parameters and output artifacts | no | no | yes |
| Ability to run experiment on remote machine with modified parameters | no | no | yes |