Australian Team Running Its Own DARPA-Style Cave Challenge to Test Robots
Although the in-person Systems Track event of the DARPA SubT Challenge was cancelled because of the global pandemic, the Systems Track teams still have to prepare for the Final Event in 2021, which will include a cave component. Systems Track teams have been on their own to find cave environments to test in, and many of them are running their own DARPA-style competitions to test their software and hardware.
We'll be posting a series of interviews exploring where and how the teams are making this happen, and today we're featuring Team CSIRO Data 61, based in Brisbane, Australia.
This interview features the following roboticists from Team CSIRO Data 61:
Katrina Lo Surdo-Electrical and Computer Engineer, Advanced Mechatronics Systems
Nicolas Hudson-Senior Principal Research Scientist, Group Leader
Navinda Kottege-Principal Research Scientist, Dynamic Platforms Team Leader
Fletcher Talbot-Software Engineer, Dynamic Platforms and Primary Robot Operator
IEEE Spectrum: Tell me about your cave! How'd you find your cave, and what kind of cave was it?
Katrina Lo Surdo: We basically just sent a bunch of emails around to different caving clubs all across Australia asking if they knew where we could test our robots, and most of them said no. But this particular caving club in Chillagoe (a 20 hours' drive north of Brisbane) said they knew of a good cave. The caves in Chillagoe used to be coral reefs-they were formed about 400 million years ago, and then over time the reefs turned into limestone and then that limestone eroded into caves. In the particular cave that we went to, although a lot of the formations and the actual sort of caverns themselves are formed by limestone, there's a lot of sediment that has been deposited inside the caves so the floor is reasonably flat. And it's got that red dirt feel that you think of when you think of Australia.
I do think this cave had a good mix of a lot of the elements that most caves would have. It did have some verticality, some massive caverns, and some really small constrained passageways. And it was really sprawling as well, so I think it was a good representation of a lot of different types of caves.
Were you looking for any cave you could find, or a cave that was particularly robot friendly?
Lo Surdo: We wanted to be able to succeed as much as possible, but the cave needed to provide enough of a challenge that it would be useful for us to go. So if it was going to be completely flat with no obstacles, I don't think it would have been good. And another thing that would be looked at was whether the cave itself is fragile or anything, because obviously we're rolling our robots around and we don't want to be damaging it.
The terrain itself was quite extreme, although a human could walk through a large portion of it without difficulty.
Nicolas Hudson: We should add that Katrina is an experienced caver and an expert climber, so when she says it's easily traversable by a human, she means that cavers find it easy. There were others on the team who were not comfortable at all in the cave.
What do you feel like the biggest new challenge was, going from an urban environment to a cave environment?
Hudson: My take going from the Urban Circuit to this cave was that at Urban, it was essentially set up so that a human, legged, or tracked system could traverse the entire thing. For example, at Urban, we flew a drone through a hole in the floor, but there was a staircase right next to it. In the cave, there were parts that were only drone-accessible.
Another good example is that our drone actually flew way beyond the course we expected at one point, because we don't have any artificial constraints-it's just the cave system. And it was flying through an area that we weren't comfortable going as people. So, I think the cave system was really a place where the mobility of drones shines in certain areas even more so than urban environments. That was the most important difference from my perspective.
How did your team of robots change between Urban and Cave?
Hudson: Our robots didn't change a lot. We kept the large Titan robots because they're by far our most capable ground platform. In my opinion, they're actually more capable than legs on slippery intense slopes because of the amount of grip that they have. There are things I wouldn't walk up that the Titan can drive up. So that stayed as our primary platform.
While the larger platforms could cover a lot of ground and were very stable, the smaller tracked platforms, SuperDroid robots which are about a meter long, didn't even function in the cave. Like, they went a meter and then the just traction wasn't enough, because they were too small. We've started working a beefed-up small tracked platform that has a lot more grip. We decided not to push for legs in the cave. We have a Ghost Vision 60. And we thought about, do we go legged in this environment, and we decided not to because of how unstructured it was, and just because of the difficulty of human traversing it.
I really think the big difference was the drone played a much larger role. Where in Urban the drone had this targeted investigation role where it would be sitting on the back of the Titans and it would take off and you'd send it up through a hole or something like that, in the cave, what we found ourselves doing was really using it to sort of scout because the ground was just so challenging. The cost to go 20 meters in a cave with a ground robot can be absurdly difficult. And so getting better situational awareness quickly with the drone was probably where the concept of operations changed more than the robots did.
Photo: Team CSIRO Data 61 With such extreme mobility challenges, why use ground robots at all? Why not just stick with drones?
Hudson: We found that perception was significantly better on the ground robots. The ground robots have four cameras, and so they're running 360 vision the whole time for object detection. The drone was great as a scout, but it was really difficult for it to find objects because there are so many crevices that to look through every area with a drone is very time consuming and they run out of battery. And so it's really the endurance of the ground robot and the better perception where they played their part.
We used the drones to figure out the topological layout of the cave. We didn't let the operators see the cave beforehand, and it's sort of hard to comprehend-in Urban, the drone did quite well because there were these very geometric rooms and so you could sort of cover things with a gimbal camera. But in the cave there's just so many strange structures, and you have very poor camera coverage with a single camera.
When you're using the drones and the ground robots together, how are the robots able to decide where it's safe to go with that terrain variability?
Hudson: I'll answer that with respect to our first couple mock-competition runs, where the robot operators didn't have any prior knowledge of the cave. What happened is that once the drones did a scouting mission, the operator gets a reasonably good idea if there's any constrictions or any large elevation changes. And then we spread out the ground robots to different areas and tried things.
Our autonomy system went up some things we didn't expect it to-we just thought it would say don't go there." And in other cases there was a little ledge or a series of rocks that the autonomy system said I don't want to do that" but it looked traversable in the map. We have a sort of backup teleoperation mode where you can just command the velocity of the tracks. One time, that was beneficial, in that it actually went through something that the autonomy system didn't. But the other two times, it ended up flipping the robot, and one of those times, it actually flipped the robot and crushed the drone.
So a real lesson learned is that it was incredibly hard for human operators to perceive what was traversable and what was not, even with 3D point clouds and cameras. My overwhelming impression was it was unbelievably difficult to predict, as a person, what was traversable by a robot.
Lo Surdo: And the autonomy did a much better job at choosing a path.
So the autonomy was doing a better job than the human teleoperators, even in this complex environment?
Hudson: It's a difficult question to answer. Half of the time, that's absolutely correct: The robot was more capable than the human thought it would be. There were other times that I think a human with a teleoperation system standing right next to the robot could better understand things like crazy terrain formations or dust, and the robot just didn't have that context. I think if I had to rank it, a person with a remote control right next to the robot is probably the gold standard. We never really had issues with that. Then but the autonomy was definitely better than someone at the base station with a little bit of latency.
And that's much different than your experience with Tunnel or Urban, right? Where a human teleoperator could be both more efficient and safer than a fully autonomous robot?
Hudson: That's right.
What were some challenges that were unique to the cave?
Hudson: The cave terrain was a big mix of things. There was a dry river bed in parts of it, and then other parts of it had these rocks that look almost like coral. There were formations that drop from the ceiling, things that have grown up from the ground, and it was just this completely random distribution of obstacles that's hard for a human to make up, if that makes sense. And we definitely saw the robots getting trapped once or twice by those kinds of things.
Every run that we had we ended up with our large ground robots flipped over at least once, and that almost always occurred because it slipped off a two meter drop when the terrain deformed underneath the robot. Because the Titans are so sturdily built, the perception pack was protected, and the entire setup could be turned back over and they kept working.
Lo Surdo: There was also quite a natural flow to the terrain, because that's where people had traversed through, and I think in a lot of cases the autonomy did a pretty good job of picking its way through those obstacles, and following the path that the humans had taken to get to different places. That was impressive to me.
Navinda Kottege: I think the randomness also may be related to the relatively poor performance of the operators, because in the other SubT circuits, the level of situational awareness they got from the sensors would be augmented by their prior experience. Even in Urban, if it's a room, it's a geometric shape, and the human operator can kind of fill in the blanks because they have some prior experience. In caves, since they haven't experienced that kind of environment, with the patchy situational awareness they get from the sensors it's very challenging to make assumptions about what the environment around the robot is like.
What kind of experience did you have as a robot operator during your mock Cave Circuit competition?
Fletcher Talbot: It was extremely difficult. We made some big assumptions which turned out to be very wrong about the terrain, because myself and the other operator were completely unaware of what the cave looked like-we didn't see any photos or anything before we actually visited. And my internal idea of what it would look like was wrong initially, misinformed somewhat by some of the feedback we got back from point clouds and meshes, and then rudely awakened by going on a tour through the cave after our mock competition ended.
During our runs we saw some slopes that looked completely traversable and so we tried to send robots up those slopes-if I had known what those slopes actually looked like, I never would have done that. But the robots themselves were beasts, and just did stuff that we would never have thought possible." -Fletcher Talbot, Team CSIRO Data 61For example, during our runs we saw some slopes that looked completely traversable and so we tried to send robots up those slopes-if I had known what those slopes actually looked like, I never would have done that. But the robots themselves were beasts, and just did stuff that we would never have thought possible.
We definitely learned that operators can hamper the progress of the robots, because we don't really know what we're doing sometimes. My approach going through the different runs was to just let the robots be more autonomous, and just give them very high level commands rather than trying to do any kind of finessing into gaps and stuff. That was my takeaway- trust the autonomy.
So the cave circuit has gotten you to trust your robots more?
Talbot: Yeah, some other stuff that robots did was insane. As operators we never would have expected them to be able to do it, or commanded them to do it in the first place.
Photo: Team CSIRO Data 61 What were the results of the competition that you held?
Lo Surdo: We made our best guess as to what DARPA would do in an environment like this, and hid artifacts around the cave in the way that we've seen them hide artifacts before.
Hudson: We set up the staging area with a team of 14 people; we took a lot of people because it was only a 20-hour drive away [as opposed to a flight across the world]. The operators came in and only saw the staging area.
Talbot: We were blind to what the course was going to be like, or where the objects were. We only knew what objects were brought.
Kottege: We did four runs overall, dividing the cave into two courses, doing two runs each.
Talbot: The performance was reasonably consistent, I think, throughout all the runs. It was always four or five objects detected, about half the ones that were placed on the course.
How are you feeling about the combined circuit for the SubT Final?
Kottege: I think we have some pretty good ideas of what we need to focus on, but there's also this big question of how DARPA will set up the combined event. So, once that announcement is made, there will be some more tweaking of our approach.
This is probably true for other teams as well, but after we performed at Urban, we felt like if we got a chance to do Tunnel again, we'd be able to really ace it, because we'd improved that much. Similarly, once we did our cave testing, we've had a similar sentiment- that if we got a chance to do Urban again, we'd probably do far better. I think that's a really good place to be at, but I'm sure DARPA has some interesting challenges in mind for the final.
Lo Surdo: I do think that us going to the cave gives us a bit of an advantage, because there's some terrain that you can't really make or simulate, and some of the stuff we learned was really valuable and I think we'll really serve us in the next competition. One thing in particular was the way that our robots assessed risk-we went up some really crazy terrain which was amazing, but in some instances there was a really easy pathway right next to it. So assessing risk is something that we're going to be looking at improving in the future.
Talbot: With the cave, it's hard to gauge the difficulty level compared to what DARPA might have given us-whether we met that difficulty level or went way above it or maybe even undershot it, we don't really know. But it'll be very interesting to see what DARPA throws at us, or if they give us some indication of what they were going to give us the cave so we can sort of balance it and figure out whether we hit the mark.
Photo: Team CSIRO Data 61 Now that you've been through Tunnel and Urban and your own version of cave, do you feel like you're approaching a generalizable solution for underground environments?
Kottege: I'm fairly confident that we are approaching a generalizable state where our robots can perform quite well in a given underground environment.
Talbot: Yeah, I think we are getting there. I think it needs some refinement, but I think the key components are there. One of the benefits of doing these field trips, and hopefully we do more in the future, is that we don't really know what we can't do until we come across that obstacle in real life. And then we go, oh crap, we're not prepared for that!" But from all the test environments that we've been in, I think we have a somewhat generalizable solution.
Read more DARPA SubT coverage from IEEE Spectrum