- Article
Learn how to develop a training script with a notebook on an Azure Machine Learning cloud workstation. This tutorial covers the basics you need to get started:
- Set up and configuring the cloud workstation. Your cloud workstation is powered by an Azure Machine Learning compute instance, which is pre-configured with environments to support your various model development needs.
- Use cloud-based development environments.
- Use MLflow to track your model metrics, all from within a notebook.
Prerequisites
To use Azure Machine Learning, you'll first need a workspace. If you don't have one, complete Create resources you need to get started to create a workspace and learn more about using it.
Start with Notebooks
The Notebooks section in your workspace is a good place to start learning about Azure Machine Learning and its capabilities. Here you can connect to compute resources, work with a terminal, and edit and run Jupyter Notebooks and scripts.
Sign in to Azure Machine Learning studio.
Select your workspace if it isn't already open.
On the left navigation, select Notebooks.
If you don't have a compute instance, you'll see Create compute in the middle of the screen. Select Create compute and fill out the form. You can use all the defaults. (If you already have a compute instance, you'll instead see Terminal in that spot. You'll use Terminal later in this tutorial.)
Set up a new environment for prototyping (optional)
In order for your script to run, you need to be working in an environment configured with the dependencies and libraries the code expects. This section helps you create an environment tailored to your code.To create the new Jupyter kernel your notebook connects to, you'll use a YAML file that defines the dependencies.
Upload a file.
Files you upload are stored in an Azure file share, and these files are mounted to each compute instance and shared within the workspace.
Download this conda environment file, workstation_env.yml to your computer.
Select Add files, then select Upload files to upload it to your workspace.
Select Browse and select file(s).
Select workstation_env.yml file you downloaded.
Select Upload.
You'll see the workstation_env.yml file under your username folder in the Files tab. Select this file to preview it, and see what dependencies it specifies.
Create a kernel.
Now use the Azure Machine Learning terminal to create a new Jupyter kernel, based on the workstation_env.yml file.
Select Terminal to open a terminal window. You can also open the terminal from the left command bar:
If the compute instance is stopped, select Start compute and wait until it's running.
Once the compute is running, you see a welcome message in the terminal, and you can start typing commands.
View your current conda environments. The active environment is marked with a *.
conda env list
If you created a subfolder for this tutorial,
cd
to that folder now.Create the environment based on the conda file provided. It takes a few minutes to build this environment.
conda env create -f workstation_env.yml
Activate the new environment.
conda activate workstation_env
Validate the correct environment is active, again looking for the environment marked with a *.
conda env list
Create a new Jupyter kernel based on your active environment.
python -m ipykernel install --user --name workstation_env --display-name "Tutorial Workstation Env"
Close the terminal window.
You now have a new kernel. Next you'll open a notebook and use this kernel.
Create a notebook
Select Add files, and choose Create new file.
Name your new notebook develop-tutorial.ipynb (or enter your preferred name).
If the compute instance is stopped, select Start compute and wait until it's running.
You'll see the notebook is connected to the default kernel in the top right. Switch to use the Tutorial Workstation Env kernel.
Develop a training script
In this section, you develop a Python training script that predicts credit card default payments, using the prepared test and training datasets from the UCI dataset.
This code uses sklearn
for training and MLflow for logging the metrics.
Start with code that imports the packages and libraries you'll use in the training script.
import osimport argparseimport pandas as pdimport mlflowimport mlflow.sklearnfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.metrics import classification_reportfrom sklearn.model_selection import train_test_split
Next, load and process the data for this experiment. In this tutorial, you read the data from a file on the internet.
# load the datacredit_df = pd.read_csv( "https://azuremlexamples.blob.core.windows.net/datasets/credit_card/default_of_credit_card_clients.csv", header=1, index_col=0,)train_df, test_df = train_test_split( credit_df, test_size=0.25,)
Get the data ready for training:
# Extracting the label columny_train = train_df.pop("default payment next month")# convert the dataframe values to arrayX_train = train_df.values# Extracting the label columny_test = test_df.pop("default payment next month")# convert the dataframe values to arrayX_test = test_df.values
Add code to start autologging with
MLflow
, so that you can track the metrics and results. With the iterative nature of model development,MLflow
helps you log model parameters and results. Refer back to those runs to compare and understand how your model performs. The logs also provide context for when you're ready to move from the development phase to the training phase of your workflows within Azure Machine Learning.# set name for loggingmlflow.set_experiment("Develop on cloud tutorial")# enable autologging with MLflowmlflow.sklearn.autolog()
Train a model.
# Train Gradient Boosting Classifierprint(f"Training with data of shape {X_train.shape}")mlflow.start_run()clf = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)clf.fit(X_train, y_train)y_pred = clf.predict(X_test)print(classification_report(y_test, y_pred))# Stop logging for this modelmlflow.end_run()
Iterate
Now that you have model results, you may want to change something and try again. For example, try a different classifier technique:
# Train AdaBoost Classifierfrom sklearn.ensemble import AdaBoostClassifierprint(f"Training with data of shape {X_train.shape}")mlflow.start_run()ada = AdaBoostClassifier()ada.fit(X_train, y_train)y_pred = ada.predict(X_test)print(classification_report(y_test, y_pred))# Stop logging for this modelmlflow.end_run()
Examine results
Now that you've tried two different models, use the results tracked by MLFfow
to decide which model is better. You can reference metrics like accuracy, or other indicators that matter most for your scenarios. You can dive into these results in more detail by looking at the jobs created by MLflow
.
On the left navigation, select Jobs.
Select the link for Develop on cloud tutorial.
There are two different jobs shown, one for each of the models you tried. These names are autogenerated. As you hover over a name, use the pencil tool next to the name if you want to rename it.
Select the link for the first job. The name appears at the top. You can also rename it here with the pencil tool.
The page shows details of the job, such as properties, outputs, tags, and parameters. Under Tags, you'll see the estimator_name, which describes the type of model.
Select the Metrics tab to view the metrics that were logged by
MLflow
. (Expect your results to differ, as you have a different training set.)Select the Images tab to view the images generated by
MLflow
.Go back and review the metrics and images for the other model.
Create a Python script
Now create a Python script from your notebook for model training.
On the notebook toolbar, select the menu.
Select Export as> Python.
Name the file train.py.
Look through this file and delete the code you don't want in the training script. For example, keep the code for the model you wish to use, and delete code for the model you don't want.
- Make sure you keep the code that starts autologging (
mlflow.sklearn.autolog()
). - You may wish to delete the autogenerated comments and add in more of your own comments.
- When you run the Python script interactively (in a terminal or notebook), you can keep the line that defines the experiment name (
mlflow.set_experiment("Develop on cloud tutorial")
). Or even give it a different name to see it as a different entry in the Jobs section. But when you prepare the script for a training job, that line won't work and should be omitted - the job definition includes the experiment name. - When you train a single model, the lines to start and end a run (
mlflow.start_run()
andmlflow.end_run()
) are also not necessary (they'll have no effect), but can be left in if you wish.
- Make sure you keep the code that starts autologging (
When you're finished with your edits, save the file.
You now have a Python script to use for training your preferred model.
Run the Python script
For now, you're running this code on your compute instance, which is your Azure Machine Learning development environment. Tutorial: Train a model shows you how to run a training script in a more scalable way on more powerful compute resources.
On the left, select Open terminal to open a terminal window.
View your current conda environments. The active environment is marked with a *.
conda env list
Activate your kernel:
conda activate workstation_env
If you created a subfolder for this tutorial,
cd
to that folder now.Run your training script.
python train.py
Examine script results
Go back to Jobs to see the results of your training script. Keep in mind that the training data changes with each split, so the results differ between runs as well.
Clean up resources
If you plan to continue now to other tutorials, skip to Next steps.
Stop compute instance
If you're not going to use it now, stop the compute instance:
- In the studio, in the left navigation area, select Compute.
- In the top tabs, select Compute instances
- Select the compute instance in the list.
- On the top toolbar, select Stop.
Delete all resources
Important
The resources that you created can be used as prerequisites to other Azure Machine Learning tutorials and how-to articles.
If you don't plan to use any of the resources that you created, delete them so you don't incur any charges:
In the Azure portal, select Resource groups on the far left.
From the list, select the resource group that you created.
Select Delete resource group.
Enter the resource group name. Then select Delete.
Next steps
Learn more about:
- From artifacts to models in MLflow
- Using Git with Azure Machine Learning
- Running Jupyter notebooks in your workspace
- Working with a compute instance terminal in your workspace
- Manage notebook and terminal sessions
This tutorial showed you the early steps of creating a model, prototyping on the same machine where the code resides. For your production training, learn how to use that training script on more powerful remote compute resources:
Train a model
FAQs
How do I deploy a machine learning model in Azure cloud? ›
- Register the model.
- Prepare an entry script.
- Prepare an inference configuration.
- Deploy the model locally to ensure everything works.
- Choose a compute target.
- Deploy the model to the cloud.
- Test the resulting web service.
- Sign in to Azure Machine Learning studio.
- Select Create workspace.
- Provide the following information to configure your new workspace: Field. Description. Workspace name. Enter a unique name that identifies your workspace. ...
- Select Create to create the workspace.
Azure Machine Learning provides several ways to train your models, from code-first solutions using the SDK to low-code solutions such as automated machine learning and the visual designer.
How do I create an Azure model? ›In server Overview, click New model. In New model > Choose a datasource, verify Sample data is selected, and then click Add. In Overview, verify the adventureworks sample model is added. Note: You can also choose the dataset from SQL data warehouse/azure SQLDB or from powerBI file.
What is model in Azure ML? ›A model is the result of a Azure Machine learning training Run or some other model training process outside of Azure. Regardless of how the model is produced, it can be registered in a workspace, where it is represented by a name and a version.
What are the 4 basic steps for deploying a virtual machine in Azure? ›Step 1 − Login to Azure Management Portal. Step 2 − Locate and click on 'Virtual Machines' in the left panel and then click on 'Create a Virtual Machine'. Step 3 − Alternatively, click 'New' at the bottom left corner and then click 'Compute' → 'Virtual Machine' →'Quick Create'. Step 4 − Enter DNS name.
How do you create a machine learning model on the cloud? ›- On this page.
- Before you begin.
- Store your model in Cloud Storage. Set up your Cloud Storage bucket. Upload the exported model to Cloud Storage. Upload custom code.
- Test your model with local predictions.
- Deploy models and versions. Create a model resource. Create a model version.
- Step 1: Create a new virtual environment using Pycharm IDE.
- Step 2: Install necessary libraries.
- Step 3: Build the best machine learning model and Save it.
- Step 4: Test the loaded model.
- Step 5: Create main.py file.
The Azure Machine Learning studio lets you quickly create a workspace with default settings. Use Azure portal for a point-and-click interface with more security options. Use the VS Code extension if you work in Visual Studio Code.
How do I create a dataset in Azure machine learning? ›- Verify that you have contributor or owner access to the underlying storage service of your registered Azure Machine Learning datastore. Check your storage account permissions in the Azure portal.
- Create the dataset by referencing paths in the datastore.
Does Azure ML require coding? ›
Azure Machine Learning designer: use the designer to train and deploy machine learning models without writing any code.
What programming languages does Azure ML support? ›On the other hand, Azure Machine learning service supports Python, R and . NET programming languages. And if you are working with Python then you can use some common frameworks like — PyTorch, TensorFlow, LightGBM and scikit-learn.
Is Azure ML easy to learn? ›Fast and easy Machine Learning product
Microsoft Azure Machine Learning provides highest availability and is very pocket friendly for any sized company. Its intelligent bot service provides great customer service by interacting them with very high speed.
The three cloud computing service models are IaaS, PaaS, and SaaS. You can also use serverless computing to eliminate the need to manage infrastructure.
How do you train a model in Azure? ›- Connect to the workspace. Tip. Use the tabs below to select the method you want to use to train a model. ...
- Create a compute resource for training. An Azure Machine Learning compute cluster is a fully managed compute resource that can be used to run the training job. ...
- Submit the training job. Python SDK.
- Enter virtual machines in the search.
- Under Services, select Virtual machines.
- In the Virtual machines page, select Create and then Virtual machine. ...
- In the Basics tab, under Project details, make sure the correct subscription is selected and then choose to Create new resource group.
Amazon ML supports three types of ML models: binary classification, multiclass classification, and regression.
What are the three ML models? ›- Descriptive - to help understand what happened in the past.
- Prescriptive - to automate business decisions and processes based on data.
- Predictive - to predict future business scenarios.
With model registration, you can store and version your models in the Azure cloud, in your workspace. The model registry makes it easy to organize and keep track of your trained models. A registered model is a logical container for one or more files that make up your model.
How do you deploy a ML model in Azure function? ›- In Solution Explorer, right-click on your project and select Manage NuGet Packages.
- Choose "nuget.org" as the Package source.
- Select the "Browse" tab.
- Search for Microsoft.ML.
- Select that package in the list, and select the Install button.
- Select the OK button on the Preview Changes dialog.
How many types of VMs are there in Azure? ›
Azure VM Types
Azure groups VMs into five categories, detailed in the table below. Provide a balanced ratio of CPU-to-memory. These VMs are apt for testing and development environments, small to medium databases, and low to medium traffic web servers.
There are three different ways to deploy cloud services: on a public cloud, private cloud or hybrid cloud.
What are the 7 key steps to build your machine learning model? ›- Collecting Data: As you know, machines initially learn from the data that you give them. ...
- Preparing the Data: After you have your data, you have to prepare it. ...
- Choosing a Model: ...
- Training the Model: ...
- Evaluating the Model: ...
- Parameter Tuning: ...
- Making Predictions.
- Work backwards from the customer.
- Re-envision the world as products.
- Organize the teams around products.
- Bring the work to the teams.
- Reduce risk through iteration and automation.
- Own your own lifecycle.
In this article. A machine learning model is a file that has been trained to recognize certain types of patterns. You train a model over a set of data, providing it an algorithm that it can use to reason over and learn from those data.
How to build a machine learning model in 10 minutes? ›- 10 Minutes to Building a Machine Learning Pipeline with Apache Airflow. ...
- Sign in to Google Cloud Platform and Create a Compute Instance. ...
- Pull the Trained Model from Github. ...
- Overview of ML Pipeline components. ...
- Install Docker and Set up Virtual Hosts Using nginx. ...
- Build and Run Docker Container.
Machine learning models are created by training algorithms with either labeled or unlabeled data, or a mix of both.
How long does it take to build a ML model? ›What goes into creating a machine learning model. , 50% of respondents said it took 8–90 days to deploy one model, with only 14% saying they could deploy in less than a week.
What are the 4 steps to make a machine learn? ›- Stage 1: Collect and prepare data. ...
- Stage 2: Make sense of data. ...
- Stage 3: Use data to answer questions. ...
- Stage 4: Create predictive applications.
- IBM Watson Studio.
- Databricks Lakehouse Platform.
- MATLAB.
- Alteryx.
- SAS Visual Data Mining and Machine Learning.
- TensorFlow.
- Anaconda.
- Deepnote.
How do I Create an Azure virtual machine template? ›
- Select the following image to sign in to Azure and open a template. The template creates a key vault and a secret.
- Select or enter the following values. Use the default values, when available. ...
- Select Review + create. After validation completes, select Create to create and deploy the VM.
- 1: Set up your local development environment.
- 2: Install the needed Azure library packages.
- 3: Write code to create a virtual machine.
- Run the script.
- Verify the resources.
- 6: Clean up resources.
- See also.
- Go to the Azure portal to find a managed image. ...
- Select the image you want to use from the list. ...
- Select Create VM from the menu.
- Enter the virtual machine information. ...
- Select a size for the VM. ...
- Under Settings, make changes as necessary and select OK.
Represents a resource for exploring, transforming, and managing data in Azure Machine Learning. A Dataset is a reference to data in a Datastore or behind public web urls.
How do I create a data flow in Azure? ›To create a data flow, select the plus sign next to Factory Resources, and then select Data Flow. This action takes you to the data flow canvas, where you can create your transformation logic. Select Add source to start configuring your source transformation.
How do I create a data science virtual machine in Azure? ›- Go to the Azure portal You might be prompted to sign in to your Azure account if you're not already signed in.
- Find the virtual machine listing by typing in "data science virtual machine" and selecting "Data Science Virtual Machine - Windows 2019."
- Select the Create button at the bottom.
Microsoft Azure is widely considered both a Platform as a Service (PaaS) and Infrastructure as a Service (IaaS) offering.
Do you need to know Python for Azure? ›Yes, you can learn Microsoft Azure without learning to program. But this would restrict your work roles to just non-technical roles. If you're a beginner, learning a programming language is recommended, such as Python, to get a high-paying job in the cloud computing industry.
Can a non coder learn Azure? ›You do not need coding skills to use Microsoft Azure.
The Microsoft Azure web portal provides all the functionality you need to manage your cloud infrastructure without previous coding experience.
Azure Developer salary in India ranges between ₹ 4.0 Lakhs to ₹ 12.4 Lakhs with an average annual salary of ₹ 6.5 Lakhs.
What is the most popular language on Azure? ›
Azure supports the most popular programming languages in use today, including Python, JavaScript, Java, . NET and Go.
What is the best programming language to learn for Azure? ›Most people would say that the best programming language for Azure is Node. js because it is a very easy-to-learn and powerful language. However, C# is also a better choice to learn if you want to have a career in enterprise development.
What are limitations of Azure ML? ›Resource | Maximum limit |
---|---|
Datasets | 10 million |
Runs | 10 million |
Models | 10 million |
Artifacts | 10 million |
Average Microsoft Corporation Machine Learning Engineer salary in India is ₹ 42.9 Lakhs for 4 years of experience.
What is the salary of Azure ML certification? ›Average salary for a Azure Data Engineer in India is 6.5 Lakhs per year (₹54.2k per month).
How do you create a model in machine learning? ›- In the Amazon ML console, choose Amazon Machine Learning, and then choose ML models.
- On the ML models summary page, choose Create a new ML model.
- If you have already created a datasource, on the Input data page, choose I already created a datasource pointing to my S3 data.
- Contextualise machine learning in your organisation.
- Explore the data and choose the type of algorithm.
- Prepare and clean the dataset.
- Split the prepared dataset and perform cross validation.
- Perform machine learning optimisation.
- Deploy the model.
- In Solution Explorer, right-click on your project and select Manage NuGet Packages.
- Choose "nuget.org" as the Package source.
- Select the "Browse" tab.
- Search for Microsoft.ML.
- Select that package in the list, and select the Install button.
- Select the OK button on the Preview Changes dialog.
- Define the purpose of your model, the problem you are trying to solve, or the story you are trying to tell. ...
- Determine the model boundary. ...
- Map the model. ...
- Build the model. ...
- Test the model. ...
- Create an interface. ...
- Share the model.
The ML model development involves data acquisition from multiple trusted sources, data processing to make suitable for building the model, choose algorithm to build the model, build model, compute performance metrics and choose best performing model.
What format is a machine learning model? ›
Windows Machine Learning uses the Open Neural Network Exchange (ONNX) format for its models. You can download a pre-trained model, or you can train your own model.
What are the main service models in Azure? ›The three cloud computing service models are IaaS, PaaS, and SaaS. You can also use serverless computing to eliminate the need to manage infrastructure. The shared responsibility model determines the security tasks that are handled by the cloud provider and handled by the customer.
What are the different Azure service models? ›- Microsoft Sentinel.
- Protect your Azure resources from distributed denial-of-service (DDoS) attacks.
- Azure Bastion. Fully managed service that helps secure remote access to your virtual machines.
- Web Application Firewall. ...
- Azure Firewall. ...
- Azure Firewall Manager.
Azure Object Anchors is a managed cloud service that converts 3D models into AI models that enable object-aware mixed reality experiences for the HoloLens.
What programming language does Azure use? ›Azure supports the most popular programming languages in use today, including Python, JavaScript, Java, . NET and Go.
How do I add Azure Machine Learning in Excel? ›All you need to do is add the web service by providing the URL (found on the API help page) and the API key (found on the API dashboard page). When you save the Excel workbook, your web services are also saved, so you can share the workbook with others and enable them to use the web service.
Can you use TensorFlow in Azure? ›Azure Machine Learning supports running distributed TensorFlow jobs with both Horovod and TensorFlow's built-in distributed training API. For more information about distributed training, see the Distributed GPU training guide.