How well can we replicate complex, non-standard, human-like movements with the flight stick?

[False Prophet]

I wonder if we have a flight stick or the "ball" (as in the 0093 U.C EFF cockpit), can we make a humanoid machine do complex movements such as the "Char's kick" if those movements have never been programmed into the machine? I guess that the onboard computer will have to do a lot of heavy work in interpreting the pilot's inputs into several smaller steps to perform. I recently read a Twitter thread about how when programming games for the Wii, you have to factor in how would the Wii compensate for different body type, health status, etc. It is how old people still did so well in Wii Sport.

[MythSearcher]

Motion module, basically just have standard motion with computer analysis the distance from the target and judge what interaction should be performed.
Action can be separated into aggressive and non-aggressive and managed by just 2 buttons, but you can always give it a little more buttons. If you want complexity, you can actually use something like the game pad, using the Dual Shock of the PS family as an example, you get 4 interaction buttons(○, X, □, △), then you can use the R1,2 and L1,2 as shift keys, so basically you get 20 different interactions.
Then you need to account for the 3 directional movement and 3 rotational movement.

It can be something like a fighting game, which is exactly the same idea.
You have less freedom, but if you are familiar with it, you can get pretty well.

Char's kick is likely just a custom action, say you can set up L1+○ as the kick and the computer will adjust the angle and acceleration for you. Just like the victory sign by Keith in 0083 final scene.

[SonicSP]

If it's not pre-programmed in, I don't think the Char kick can be done unless the segmented actions of that kick is already in and you just chain them together.

While never directly elaborated, I always assumed the cloud-based "Veda Backup" in 00 refers to stuff like this since it's directly stated in that show that the strain on the pilot increased after they switched to the Stand Alone OS. Either slower reaction time by the inferior software interpreter, just worse interpretation in general, etc.

As a FIFA player (yes the football game), I love the "Auto" setting of things like passes and shooting and it really does reduce the strain on your piloting so to speak, as things like direction and power are more automated. However Auto also has the computer more likely misinterprets your intentions like on which player to pass to, or how much power you intend, etc. It can't read your mind.

Which is why different players use Semi or even full Manual for certain actions in that game cos they prefer having more control even if it means more strain. It's based on how much you want to delegate to the computer, your own skillets, cost-benefit analysis.....and how good the computer is for that action type in that year's game version.

[Seto Kaiba]

It's all a matter of them being programmed into the control computer beforehand.

Most of a Mobile Suit's maneuvering is contextual, based on the inputs from the controls combined with the MS's own AMBAC and other systems understanding of the MS's posture.

[Underrated GM Custom]

While never directly elaborated, I always assumed the cloud-based "Veda Backup" in 00 refers to stuff like this since it's directly stated in that show that the strain on the pilot increased after they switched to the Stand Alone OS. Either slower reaction time by the inferior software interpreter, just worse interpretation in general, etc.

Something similar occurs in Sentinel, going off memory here so the terms won't be exact. There is a software time bomb so to speak that messes with some of the preprogrammed routines a mobile suit. They did a good job with Chekhov's gun by having the instructor corps talk about how their manual piloting for landing was being traced and uploaded so that all pilots would have perfect preprogrammed landing routines. One the software bomb goes off it makes it quite a bit harder to pilot.

[Mafty]

CCA might help in that; we actually see Quess train with the controls of the Hobby Hizak, and she's actually able to do complex gymnastic maneuvers. The only controls seem to be two Flight Sticks; maybe one is for the hands, the other is for the feet, and the buttons are probably weapons.

A bit off subject; but I almost wonder if a flight stick wouldn't be the best way to make the different parts of a mech move. Looking at Pacific Rim (yes I know Eva and others predated this concept) maybe some kind of Motion Capture thing, where the pilots movements are mimicked to the machines movements could work.

[SonicSP]

Motion capture might be too dangerous on something that moves fast. Would be harder to strap the pilot in and keep them as still as possible.

[Seto Kaiba]

CCA might help in that; we actually see Quess train with the controls of the Hobby Hizak, and she's actually able to do complex gymnastic maneuvers. The only controls seem to be two Flight Sticks; maybe one is for the hands, the other is for the feet, and the buttons are probably weapons.

Nope, the few official explanations of how MS controls work point to no direct control over the limbs at all. It's all handled by the software in the MS's computer and the AMBAC system. The whole affair apparently operates rather like driving a tank, albeit one that can maneuver in three directions. The control configuration is customizable to some degree, but the standard setup apparently uses the foot pedals as a throttle and brake, and the two sticks as configurable control levels for pitch/yaw/trim control, with various buttons on them used to navigate through the onscreen menus of the MS's interface, select weapons and so on, and fire them.

The pilot's basically just setting direction and speed/acceleration, and the MS's computer brain is doing all of the math (literally and figuratively) about where to place limbs for things like achieving the right direction of travel, walking, running, aiming weapons, etc.

A bit off subject; but I almost wonder if a flight stick wouldn't be the best way to make the different parts of a mech move. Looking at Pacific Rim (yes I know Eva and others predated this concept) maybe some kind of Motion Capture thing, where the pilots movements are mimicked to the machines movements could work.

That's called a Master-Slave system, which can be done a few different ways.

Mobile Fighter G Gundam's Mobile Trace system is an extreme, and physics-defying, example of that type of control setup where the Mobile Suit is tracking and precisely copying the movements of the pilot's limbs using sensors tied into a special suit.

Full Metal Panic! has a more workable version called a semi-master-slave motion trace system where the pilot is sitting down and restrained, but the pilot's head is in a helmet with sensors to track its orientation and their limbs are in control armatures that allow the mecha to precisely replicate their movements to a calibratable degree. (In FMP!, the cramped cockpits mean that they use a multiplier on the measured motions to achieve the desired range of motion, so even a small motion of the pilot's limbs can produce a much larger motion of the mecha's limbs and achieve even movements impossible for a flesh-and-blood human to make. Elite AS pilots like the protagonist Sousuke use a multiplication factor of 3.5x, so if they raise their arm 10 degrees, the AS would raise its arm 35 degrees.

The problem, of course, is that this level of manual control is really draining on the pilot and requires a LOT of precision and attention to detail because all the limbs are controlled manually and the mecha's balance is now entirely in the hands of the pilot's own sense of balance. Something like an ill-timed sneeze can send a mecha cartwheeling if done wrong. (See FMP! episode "A Cat and Kitten's Rock and Roll" for how.)

That's why most mecha anime cheat and follow the Gundam model, where there's a super-advanced high end computer managing things like balance and limb positioning so the pilot can focus on situational awareness.

[MythSearcher]

Full Metal Panic! has a more workable version called a semi-master-slave motion trace system where the pilot is sitting down and restrained, but the pilot's head is in a helmet with sensors to track its orientation and their limbs are in control armatures that allow the mecha to precisely replicate their movements to a calibratable degree. (In FMP!, the cramped cockpits mean that they use a multiplier on the measured motions to achieve the desired range of motion, so even a small motion of the pilot's limbs can produce a much larger motion of the mecha's limbs and achieve even movements impossible for a flesh-and-blood human to make. Elite AS pilots like the protagonist Sousuke use a multiplication factor of 3.5x, so if they raise their arm 10 degrees, the AS would raise its arm 35 degrees.

The problem, of course, is that this level of manual control is really draining on the pilot and requires a LOT of precision and attention to detail because all the limbs are controlled manually and the mecha's balance is now entirely in the hands of the pilot's own sense of balance. Something like an ill-timed sneeze can send a mecha cartwheeling if done wrong. (See FMP! episode "A Cat and Kitten's Rock and Roll" for how.)

That's why most mecha anime cheat and follow the Gundam model, where there's a super-advanced high end computer managing things like balance and limb positioning so the pilot can focus on situational awareness.

More workable but still pretty impractical.
First, things that can be done by a simple press of a button still required movement of the pilot in the Master-Slave system, meaning it will be much more exhausting.
The multiplication factor also has its own non-workable reasons. For example, when you want to raise your right hand to point the gun to your left, you body will be in you own hand's way but the AS's body won't be. And then there is this angle discrepancy. Your hand will likely be pointing in a completely different direction after a few movements were done in different axis and you need to compensate for different angles in the axis. Also, if you moved your hand in a larger angle than the AS can make, say, using a 3x multiplication but you moved your arm from the front to the side so a 90 degree motion involuntarily(because of shock from impact or something), the AS's arm will not be able to do a 270 degree motion and now both your arm and its arm are in an awkward position and you need time to adjust it back to something workable.

It is also completely not intuitive meaning no matter how much you trained, you will make mistakes once in a more panicky situation.(ironic as the title says) To make the problem more serious, they have the pilots trained as infantry at the same time, so they are even less likely to be able to reflectively use the correct motion when pushed to the limit.

So in the end they will still need a super computer to do all of these adjustments if the system has to be mass produced and used by many pilots and not just some anime heroes who can adapt to everything you throw at them and just claim they can do it.

[Seto Kaiba]

More workable but still pretty impractical.
First, things that can be done by a simple press of a button still required movement of the pilot in the Master-Slave system, meaning it will be much more exhausting.

It is noted, even in that setting, to be a very demanding system that is used mainly because it offers the greatest possible degree of versatility in robots that are using an eclectic mix of 90's-era computer technology and a lot of anachronistically advanced black technology that wouldn't/shouldn't exist in that timeframe if it were not for the temporal meddling of the whisperer.

That said, they do try to work around that by making many of the accessory and secondary features operated by voice recognition.

The multiplication factor also has its own non-workable reasons. For example, when you want to raise your right hand to point the gun to your left, you body will be in you own hand's way but the AS's body won't be. And then there is this angle discrepancy. Your hand will likely be pointing in a completely different direction after a few movements were done in different axis and you need to compensate for different angles in the axis. Also, if you moved your hand in a larger angle than the AS can make, say, using a 3x multiplication but you moved your arm from the front to the side so a 90 degree motion involuntarily(because of shock from impact or something), the AS's arm will not be able to do a 270 degree motion and now both your arm and its arm are in an awkward position and you need time to adjust it back to something workable.

It actually works pretty well in some modern robotics applications... the issues you describe are less severe than you're thinking because all the motions are relative to a neutral "zero" position, which makes it more difficult to lose track of the limb's actual orientation, though it is still noted that high bilateral factors like the ones used by the main characters are For Pros and most people don't go that high.

It is also completely not intuitive meaning no matter how much you trained, you will make mistakes once in a more panicky situation.(ironic as the title says)

That's true for any interface, though.

So in the end they will still need a super computer to do all of these adjustments if the system has to be mass produced and used by many pilots and not just some anime heroes who can adapt to everything you throw at them and just claim they can do it.

But a less super computer than you'd need to do the kind of pre-processing and sensor fusion needed to pull off the minimalist control setups used in Gundam, Macross, and many other anime titles that copycat them.

[MythSearcher]

It is noted, even in that setting, to be a very demanding system that is used mainly because it offers the greatest possible degree of versatility in robots that are using an eclectic mix of 90's-era computer technology and a lot of anachronistically advanced black technology that wouldn't/shouldn't exist in that timeframe if it were not for the temporal meddling of the whisperer.

That said, they do try to work around that by making many of the accessory and secondary features operated by voice recognition.

Surprising that they didn't have super computers given by the whisperer.
Though basically Al is a super computer AI, most of the other AS don't have one.(at least seemingly as powerful)
Voice recognition to combat level of precision likely still need very high level of processing power though.

It actually works pretty well in some modern robotics applications... the issues you describe are less severe than you're thinking because all the motions are relative to a neutral "zero" position, which makes it more difficult to lose track of the limb's actual orientation, though it is still noted that high bilateral factors like the ones used by the main characters are For Pros and most people don't go that high.

Less severe, but still quite limiting if you put it in a combat situation.
Depends on what your zero position is, things just don't go too well.
I'd assume that position is a normal resting pose, but that doesn't seem to work as the pilot have the arms resting on the body(or close to it), you have to raise the arm to the front a little before it can aim to the opposite side, but the pilot need to raise more than the AS before the arm isn't restricted by the body, so you will have more range of motions limited the higher factors you use.
I guess you can have the pilot's arms raised in the cockpit from the beginning, but then it will be at strange angles and different pilots will have to adjust for different cockpit positions as well. Especially if the high factors aren't common, the cockpits shouldn't be designed for those factors.

There is also this problem that they seem to show pilots just jumping onto other AS without having much of an issue, Sasuke should not be able to simply pilot a Savage to that kind of precision even if it takes a simple voice recognition for changing the factor. The simple difference in the mechanical parts should have enough of a variation to require fine tuning and time for adaptations.

That's true for any interface, though.

To different degrees.
You want to have the most efficient and effective system and interface.
I mean, you can probably still operate a robot with a tablet and a 3D drawing program and have the pilot drag and drop the model for movement, and the same holds true, but will be even less effective as the M-S system...
The M-S system is just, well, too strangely requiring human motion closer to regular human motion, so in instinctive moments it will trigger the intuitive responses easier. Like people tends to hold up their arms to block their faces, pretty hard to mimic with the M-S system.

Also, the more you train to be an infantry, the less reflective you can pilot an AS with those motions. Maybe it will be fine for those pros with a high factor because the motions are a lot different and they can treat it as two separate motions, but for all the other military soldiers(ahem, these should also be pros...) with a lower factor, well, you probably don't really want to have as intensive training as an infantry for your pilots. Seriously, they send them out as if they are some kind of specialised task force infantry soldiers and not specialised pilots, this is pretty much like sending out your jet fighter pilots as foot soldiers. No matter what system you are using, this by itself is strange enough.

But a less super computer than you'd need to do the kind of pre-processing and sensor fusion needed to pull off the minimalist control setups used in Gundam, Macross, and many other anime titles that copycat them.

I have doubts about this.
For Gundam and Macross, they are essentially simplified flight controls.
You need to have those auto-balancers and sensors, which likely also required by AS computers in the end anyway.
The rest is basically just have the MS controlled like a jet fighter or space craft in space, and tank on the ground with an extra jumping command. If we take out the auto-balancer, basically the motion aren't that complicated for regular computers to do, so I doubt MS and AS will have much difference in the required computer processing powers.