With the global population expected to grow from 8 billion to nearly 10 billion by 2050, farmers need to increase production by 60% to 70% on today’s arable land, believes John Deere. The company is looking to address the agriculture productivity challenge with technologies to electrify and automate its products while also helping its customers be more profitable and sustainable.
“Everything we do at John Deere is focused on real purpose and real impact,” said Jahmy Hindman, CTO at John Deere. “This means we’re developing technology that enables our customers to provide the food, fuel, fiber, and infrastructure that our growing global population needs.”
During John Deere’s CES 2023 keynote address in January, the company revealed two of its newest technologies, an electric excavator and ExactShot for planting and starter fertilizing, aimed at profitability and sustainability. At the company’s first Tech Summit in April, it elaborated on how new connectivity, autonomy, and automation technology like ExactShot can help farmers care for the land and do more with less.
ExactShot allows farmers to reduce the amount of starter fertilizer needed during planting by more than 60%. The technology uses sensors and robotics to place starter fertilizer precisely onto seeds as they are planted in the soil, rather than applying a continuous flow of fertilizer to the entire row of seeds.
The system uses a sensor to register when each seed is in the process of going into the soil. As this occurs, a robot will spray only the amount of fertilizer needed, usually about 0.2 mL, directly onto the seed at the exact moment it goes into the ground.
Across the U.S. corn crop, Deere says that ExactShot could save over 93 million gallons of starter fertilizer annually and prevent wasted fertilizer from encouraging weed growth or increasing the risk of running off the field into a waterway.
The importance of vision
ExactShot and other recent farming advances highlight how Deere has taken enormous steps in advancing vision technology in the past decade. At CES, Julian Sanchez, then Director, Emerging Technology at John Deere, provided a brief history.
In 2015, Deere launched its first vision product using a stereo vision system and geometric model trained to detect the edges of a rectangular target, helping farmers more accurately transfer crops from a harvester to an adjacent wagon. In 2016, the company released a system with a monocular camera and color detection algorithm, complemented by a GPS steering system, to detect crops in a field and ensure equipment didn’t run over them. In 2018, Deere began using CVML (computer vision and machine learning) to detect the quality of grain being harvested using a camera feeding a deep convolutional neural network inside a combine, the company’s first experience gathering large data sets to train a machine learning model.
At CES 2022, Deere unveiled a fully autonomous tractor using CVML and six pairs of stereo cameras to detect and identify objects. The technology, covered by IAV, won a 2023 CES Best of Innovation award in Robotics and was an Honoree in the Vehicle Tech & Advanced Mobility category.
The machine combines Deere’s 8R tractor, TruSet-enabled chisel plow, GPS guidance system, and new advanced technologies. The autonomous tractor became available to farmers later in the year.
Seeing and spraying
Vision also played a huge role in another Deere precision ag innovation launched in March 2022 called See & Spray Ultimate, which uses CVML to detect where every weed is and applies herbicide only and exactly where it’s needed. This is important because weeds steal valuable water, nutrients, and sun from plants, and that prevents plants from reaching their full potential.
The company had a sprayer at CES 2023 equipped with machine-vision technology. Its 120-ft carbon-fiber boom features 36 cameras to scan more than 2200 ft² of land and capture 1.2 billion pixels every second.
“If you tried to match that level of sensing and processing with human eyes, it would take nearly 6000 people,” said Sanchez.
The system performs its task while moving through uneven terrain and dusty fields and crosswinds. The cameras are sealed, ultra-rugged, and can see at night with artificial lighting as sunlight and shade.
With cameras for “eyes,” the system relies on advanced compute for “brains,” with 10 embedded controllers housing Nvidia GPUs that make sense of what the cameras see. The high-performance processors handle 4 GB of data per second while operating in harsh environments and temperatures through passive cooling.
“This system is one of the preeminent examples of cutting-edge technology, machine vision, and high performance compute at the edge operating in the harshest environments,” said Sanchez. “And because our machines are used in rural environments, where connectivity needs to be improved, edge computing enables the machine to make decisions with speed and accuracy based on the high-resolution images of every single weed and plant.”
Automating the four steps of agriculture
ExactShot, See & Spray Ultimate, and the 8R autonomous tractor are just elements of Deere’s ambitious autonomy plans for the agriculture cycle involving the four steps of tilling land, planting seeds, applying herbicides and nutrients, and harvesting crops.
“Our goal is full autonomy for all those four production cycles by the end of the decade,” said Sanchez, with one caveat. That does not include transporting the vehicles from one field to another because that requires on-road autonomy, so he calls the goal “Level 5 autonomy within field boundaries.”
“The hardest part for autonomy in agriculture is not just about going from A to B without hitting anything,” said Sanchez. “To get to full autonomy, you have to guarantee the farmer that the job that’s being done gets done right. The biggest thing that we had to do to get full autonomy was to reimagine the process.”
The pursuit of full autonomy required Deere engineers to completely rethink how they train data sets, said Sanchez. Deere started with the automotive approach of getting lots of interesting data.
“It turns out that, in agriculture, there’s just not a lot of interesting data in terms of obstacles,” said Sanchez. “Eventually we turned the problem on its head. We said we’re going to train the system to get really good at protecting things that are always there. No matter where you are, there’s always soil, there’s some kind of a tree line, and there’s some kind of a skyline. If you detect anything that doesn’t fit that model, it’s an obstacle.”
That’s how Deere engineers were able to create a breakthrough in the training algorithm standpoint for autonomy.
“We started with autonomous tillage because it’s the easiest,” said Sanchez. “There isn’t a whole lot of sensing. You drag the thing, it rips up the field, and then notice where you’ve dragged. It has GPS, so it maps where it’s been and where it needs to go.”
Other cycles of the ag production process are more challenging.
“For other jobs, like planting seed, the human is still monitoring the quality,” concluded Sanchez. “We have a way still to automate and close the loops of that job function so that you could make that fully autonomous.”
