How to control your 🚁 drone using #Azure OpenAI or OpenAI APIs and Semantic Kernel

Coding4Fun Drone 🚁 posts

Hi!

In my previous post, I wrote about how we can use ChatGPT to generate Python code to control a drone.

Today, we are going to use Azure OpenAI Services or OpenAI APIs to generate the same code using GPT models.

We are going to use the same prompt that we had for ChatGPT.

Use this python code as reference 

[CODE START]
# display battery level
sendReadCommand('battery?')
print(f'Battery: {battery} %')

# take off
sendCommand("takeoff") 

# flip the drone left
sendCommand("flip l") 

# move drone up 5 cms
sendCommand("up 5") 

# move drone left 5 cms
sendCommand("left 5") 

# rotate drone clock wise 90 degrees
sendCommand("cw 90") 

# land the drone
sendCommand("land") 

[CODE END]

Generate code to takeoff the drone, flip right, move down 30 centimeters and land

Running this prompt in Azure OpenAI Studio generates the correct Python code:

Both Azure OpenAI Services and OpenAI APIs have amazing SDKs, however, I’ll use semantic kernel to have a single code base that can use both.

Note: This is the best way to learn semantic kernel for NET and Python:

Start learning how to use Semantic Kernel.

Now, let’s create a simple function that:

Use semantic kernel to generate drone commands
Use Azure OpenAI Services or OpenAI APIs to generate drone commands
Use the semantic skill “DroneAI” to generate drone commands
Return the generated drone commands as a string

Here is the function code:

	# Copyright (c) 2023
	# Author : Bruno Capuano
	# Change Log :
	# – Use semantic kernel to generate drone commands
	# – Use Azure OpenAI Services or OpenAI APIs to generate drone commands
	# – Use the semantic skill "DroneAI" to generate drone commands
	# – Return the generated drone commands as a string
	#
	# The MIT License (MIT)
	#
	# Permission is hereby granted, free of charge, to any person obtaining a copy
	# of this software and associated documentation files (the "Software"), to deal
	# in the Software without restriction, including without limitation the rights
	# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	# copies of the Software, and to permit persons to whom the Software is
	# furnished to do so, subject to the following conditions:
	#
	# The above copyright notice and this permission notice shall be included in
	# all copies or substantial portions of the Software.
	#
	# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
	# THE SOFTWARE.

	import semantic_kernel as sk
	from semantic_kernel.connectors.ai.open_ai import AzureTextCompletion, OpenAITextCompletion

	def generate_drone_commands(user_input, use_azure_openAI=False):

	print(f'———————————————–')
	# check for empty command and add a default value
	if user_input == "":
	print(f'Empty user input, using default user input')
	user_input = "takeoff the drone, flip right, move down 30 centimeters and land"

	print(f'Input user: {user_input}')
	print(f'———————————————–')

	# create a text completion service
	kernel = sk.Kernel()

	if use_azure_openAI:
	# using Azure OpenAI Services
	print(f'>> Using Azure OpenAI Services')
	from semantic_kernel.connectors.ai.open_ai import AzureTextCompletion
	deployment, api_key, endpoint = sk.azure_openai_settings_from_dot_env()
	kernel.add_text_completion_service("dv", AzureTextCompletion(deployment, endpoint, api_key))
	else:
	# using OpenAI Services
	print(f'>> Using OpenAI Services')
	from semantic_kernel.connectors.ai.open_ai import OpenAITextCompletion
	api_key, org_id = sk.openai_settings_from_dot_env()
	kernel.add_text_completion_service("dv", OpenAITextCompletion("text-davinci-003", api_key, org_id))

	# import the semantic skill from the directory
	skill = kernel.import_semantic_skill_from_directory("skills", "DroneAI")
	drone_function = skill["Drone"]

	# generate the drone commands
	drone_command = drone_function(user_input)

	print(f'Generated Python code to control the drone')
	print(f'———————————————–')
	print(f'{drone_command}')
	print(f'———————————————–')

	return str(drone_command)

view raw DroneGenCommandsWithAI.py hosted with ❤ by GitHub

And, I need to have a Skill directory with my DroneAI skill. Something similar to this:

folder structure for a semantic kernel skill

Here is the config.json content:

{
  "schema": 1,
  "description": "Generate commands to control the drone",
  "type": "completion",
  "completion": {
    "max_tokens": 1000,
    "temperature": 0.0,
    "top_p": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0
  },
  "input": {
    "parameters": [
      {
        "name": "input",
        "description": "commands for the drone",
        "defaultValue": ""
      }
    ]
  }
}

And the skprompt.txt:

Use this python code as reference 

[CODE START]
# display battery level
sendReadCommand('battery?')
print(f'Battery: {battery} %')

# take off
sendCommand("takeoff") 

# flip the drone left
sendCommand("flip l") 

# move drone up 5 cms
sendCommand("up 5") 

# move drone left 5 cms
sendCommand("left 5") 

# rotate drone clock wise 90 degrees
sendCommand("cw 90") 

# land the drone
sendCommand("land")
[CODE END]

Generate python code only to follow these orders
{{$input}}
+++++

And that’s it! We can test this function with 2 unit tests for for platforms:

import unittest
from DroneGenCommandsWithAI import generate_drone_commands

class TestGenerateDroneCommands(unittest.TestCase):

    test_command = "takeoff the drone, flip right, move down 30 centimeters and land the drone"

    def test_generate_drone_commandsusing_openAI(self):
        print('###############################################')
        print('   Test using OpenAI APIs')
        commands = generate_drone_commands(self.test_command)

        # validate if the generated string contains the drone commands
        self.assertIn('takeoff', commands)
        self.assertIn('flip r', commands)
        self.assertIn('down 30', commands)
        self.assertIn('land', commands)
        print('###############################################')

    def test_generate_drone_commandsusing_azureOpenAI(self):
        print('###############################################')
        print('   Test using AZURE OpenAI Services')
        commands = generate_drone_commands(self.test_command, True)

        # validate if the generated string contains the drone commands
        self.assertIn('takeoff', commands)
        self.assertIn('flip r', commands)
        self.assertIn('down 30', commands)
        self.assertIn('land', commands)


if __name__ == '__main__':
    unittest.main()

And, they both works!

In the next post we will review some options and configurations necessary for the correct code generation.

Happy coding!

Greetings

El Bruno