Military Review

Technical vision and augmented reality: new research by the US Army


Experienced robot on the ground and visualization of information for the operator

Modern robotic systems are able to perform some tasks in an autonomous mode, for example, to move along a given route, taking into account the terrain and overcoming obstacles. Also, new systems are being developed that can conduct surveillance and reconnaissance, process data and provide ready information to a person. Experiments of this kind have recently taken place in the United States.

Latest Developments

New experiments and tests were carried out on August 24 by the Combat Capabilities Command and the Army Research Lab (ARL). Together with a number of other organizations, they are implementing the Artificial Intelligence for Mobility and Maneuver Essential Research Program, the goal of which is to create artificial intelligence for terrestrial RTKs and ensure its interaction with humans.

To date, within the framework of this program, several experimental models of equipment have been created. The last of them are equipped with advanced surveillance devices that provide autonomous reconnaissance and the issuance of data to a person through an augmented reality system.

For testing, two experimental RTKs were manufactured with different equipment and software, as well as a set of equipment for humans. All of these products worked at a real test site and generally proved their capabilities. Robots successfully identified suspicious objects on the ground, and operators literally saw the results of reconnaissance.

Experienced Technique

In recent tests, two RTKs of a similar appearance were used. They were built on the basis of the Warthog UGV platform from Clearpath Robotics. It is a four-wheel drive vehicle of compact dimensions with electric motors, capable of carrying loads or special equipment, incl. integrated into control loops. Such a platform is capable of operating on operator commands or in standalone mode.

Scanning lidars of two types were used to observe the situation in the experiments. One Warthog received a VLP-16 LiDAR system from Velodyne; the second was equipped with an Ouster OS1 LiDAR product. Technical vision facilities were connected with computing units and data transmission equipment.

The principle of operation of the two types of robotic scouts is quite simple. Working on the ground, the RTK must "inspect" the environment and create a three-dimensional baseline. Then the lidars continue scanning, and the electronics compares the new data with the baseline. If any changes are detected, the automation should instantly determine their nature, assess the risks and notify the person.

For the operator of such a complex, augmented reality glasses and some related equipment are intended. Information about suspicious objects and changes is displayed graphically in real time. The glasses show the mark on a specific point on the terrain and accompany it with a short description - range, degree of danger, etc. Having received the most complete information, the operator can determine further actions, both his own and the robots.

test data

Recent trials have confirmed the overall potential of the new technology. In addition, the requirements for the components have been clarified. Thus, it turned out that even simple and relatively inexpensive low-resolution lidars can effectively complement human vision. Accordingly, the need for more complex and expensive products disappears - without loss of efficiency and with some increase in productivity.

Two prototypes with different equipment

During the tests, the lidars updated the picture up to several dozen times per second. Resolution - 10 cm. This was enough for efficient work, detection of conventional targets and data delivery to the operator.

In open publications, the presence of two experimental platforms with different complexes of equipment is mentioned, but the differences that they could show during tests are not disclosed. In addition, the features of the passed tests are not specified - the type of terrain, conditional goals, the speed of the entire complex and individual systems, etc.

Directions of development

Experienced RTKs based on Warthog UGV are exclusively technology demonstrators. With their help, ARL checked the functionality of the new hardware and software for it. At the same time, the foundation is laid for the creation of fundamentally new systems that can be used in the army. On the basis of current developments in the future, new complexes of equipment can be created, suitable for implementation in practice.

The developers of the project believe that such technologies must definitely reach the army. The joint work of people and technology will allow more effective surveillance and identify more dangerous objects. RTKs will be able to find enemy ambushes, explosive devices and other threats - without endangering living soldiers.

Feedback questions need to be worked out. The operator, having received data about the found object, will be able to make his own comments and corrections, and the robot will have to process and take them into account. In addition, the creation of learning systems is not ruled out - in this case, the RTK will be able to use the accumulated experience and more correctly respond to the situation.

Another promising idea, which is being worked out in the current program, is the creation of eye protection devices with augmented reality function. The need for such technologies is considered obvious and indispensable in the context of the further development of combat equipment. Glasses can give out a wide variety of information, and not only data from a scout robot.

It should be recalled that the current work is part of the larger Artificial Intelligence for Mobility and Maneuver Essential Research Program. Its goal is to create a RTK capable of independently determining all the features and parameters of the terrain, adapting to them and freely moving and maneuvering. The proposed technical vision means can be used as a universal component for collecting all data, for driving, reconnaissance, shooting, etc.

Step by step development

To date, in the United States and other countries, robotic systems for military purposes have been created, equipped with technical vision equipment with various functions. So, they have already mastered independent movement across different areas or monitoring the situation with the identification of targets. Some of the tasks so far have to be solved using the operator's console.

One of the urgent tasks is to increase the efficiency of work in already mastered areas - to improve the "skills" for driving or more accurate target recognition. Technologies are also being created for effective interaction of the RTK with people, which is now being done by the US Army Laboratory.

In the longer term, current and future projects should lead to very interesting results. A wide range of robots of different classes and purposes will be able to appear in army units. They will accompany the soldiers on the battlefield and take on some of the tasks, as well as instantly exchange information with them. RTKs have already reduced the load on the fighters, and in the future this trend will continue.

Thus, the achievements of science and technology can again change the appearance and capabilities of the army. Apparently, the American army will be the first to master the new reconnaissance means. However, in the future, similar systems will have to appear in other countries. Time will tell what such events will lead to.
Photos used:
U.S. Army
21 comment
Dear reader, to leave comments on the publication, you must to register.

I have an account? Sign in

  1. Mavrikiy
    Mavrikiy 28 August 2020 05: 40
    RTK has already reduced the load on the fighters, and in the future this trend will continue
    Not everything is so simple. The safety of the soldier and his capabilities will increase, but the amount of information per unit of time will increase significantly + inf. noise. And how the fighter will cope is another question.
    1. Ka-52
      Ka-52 28 August 2020 06: 13
      but the amount of information per unit of time will grow several times + inf. noise. And how the fighter will cope is another question.

      therefore, the systems should carry out the primary selection of targets and issue them to the operator / user in order of priority (according to the degree of danger). Similar to the algorithms that exist in fighter airborne radars.
      1. Mavrikiy
        Mavrikiy 28 August 2020 06: 18
        Quote: Ka-52
        therefore, the systems must carry out the primary selection of targets and issue them to the operator

        I am, I am natural. request I do not mind, God help, but ..... what will this result in? To every infantryman ..... request
        1. viktor_ui
          viktor_ui 28 August 2020 08: 09
          Why does EVERY infantryman need a full-fledged vision system (if they are not autonomous combat units of the level of a pro wolfhound) ??? It will be enough for them augmented reality glasses, on which markers of the predicted operational situation will be displayed from the mobile combat control center headed by the commander. There is not and will not be any information mess after debugging such a system in practice. And what about the production of lidars in Russia and especially in terms of their mass use for their intended purpose - who has infa? And yes, technical vision for any armored box has a hefty bonus to surviving on the battlefield.
          1. psiho117
            psiho117 30 August 2020 00: 44
            Quote: viktor_ui
            What about the production of lidars in Russia

            Civilian lidars are on all unmanned taxis, and nowadays someone is not developing them.
            I know for sure that Yandex has created its own lidar for an unmanned taxi.
            As for the rest, I don't know, but under the sanctions ...
            Everything is possible bully
    2. vVvAD
      vVvAD 28 August 2020 14: 01
      Information noise should be sorted by the AI ​​itself, the correctness of the threat assessment is a matter of AI self-learning. Questions of information load and interface convenience are being worked out at the development stage.
      In any case, 2nd order (false) security alarms are better than 1st order (missed) alarms.
  2. sen
    sen 28 August 2020 06: 23
    Technical vision and augmented reality

    First of all, this is necessary for BM, MBT. Israel has a development for the Merkava tank - "sight through armor". Information from lidars should be processed by AI and displayed on the screen (or glasses) already marked (identification, degree of threat) for a person. Shooting data must be calculated for each detected target (taking into account the range, weather conditions)
  3. sergo1914
    sergo1914 28 August 2020 08: 40
    And can these scouts take off the sentries and take "tongues"?
    1. Ryaruav
      Ryaruav 28 August 2020 10: 06
      they can't shit, they can't replace a person in such matters
  4. Ryaruav
    Ryaruav 28 August 2020 10: 04
    Well, yes, a black man sitting at home in the Bronx, drinking beer and Afghanistan pulyalki
    1. psiho117
      psiho117 30 August 2020 00: 46
      black man sitting at home in the bronx drinking beer

      You have correctly described the mechanics of the action of modern aviation unmanned strike systems.
  5. Coconut
    Coconut 28 August 2020 10: 05
    Another dough
  6. Ryaruav
    Ryaruav 28 August 2020 10: 30
    the world is moving forward and the main driving force of this movement is the human brain, consciousness, a person creates machines, he works ahead of schedule, and while in the near future no machine is intellectually capable of replacing a person, I agree that the machine was mistaken and to hell with it, but still a sapper smarter
    1. vVvAD
      vVvAD 28 August 2020 14: 09
      Intellectually, yes. But by attention - no. And in terms of computing power, neither. And yet a person is more valuable. Accordingly, the task of such machines is to free a person from the constant stress and exhausting monotonous work for making key decisions, slowing down the rate of decrease in attention and increase in fatigue, maintaining efficiency and combat effectiveness for a longer time.
      And also - saving human lives, increasing situational awareness, combat effectiveness, reducing the cost of planning operations, primarily time ones, on which the success of the entire operation often depends.
    2. voyaka uh
      voyaka uh 29 August 2020 01: 30
      Computers calculate the situation and react thousands of times faster than the brain.
      Optical devices are thousands of times stronger than the eye.
  7. Momotomba
    Momotomba 28 August 2020 18: 59
    And how much will all this stuff of augmented reality weigh? And how long will the battery last? And where to get a new one after the existing set of batteries is discharged?
    1. voyaka uh
      voyaka uh 29 August 2020 01: 25
      Cartridges, mines, shells also tend to run out.
      And the supply troops regularly bring in new ones. Together with ammunition there will be
      bring up batteries and batteries.
      1. Momotomba
        Momotomba 29 August 2020 05: 49
        Quote: voyaka uh
        Together with ammunition there will be
        bring up batteries and batteries.

        It is clear that they will give you a lift and another one. The most interesting thing is to understand how much this stuff weighs. And then suddenly it will be like we have a WEARABLE COMPLEX WITH THE EFFORT OF 10 PEOPLE ...
      2. psiho117
        psiho117 30 August 2020 00: 48
        Quote: voyaka uh
        Together with ammunition there will be
        bring up batteries and batteries.

        They won't - but they are already giving a lift. American troops are quite well supplied with finger and other batteries - read the memoirs from the last wars.
  8. RealPilot
    RealPilot 29 August 2020 00: 21
    Augmented reality glasses for every infantryman? And it will be possible to catch Pokemon on them, by shooting?
    A free app from Russian / Chinese hackers in the app store will work wonders ... laughing
    1. psiho117
      psiho117 30 August 2020 00: 50
      Quite possible. They will make a civilian version on the "android" in order to recapture the loot - and catch whatever you want.
      And about the army - I doubt it.