Arckon Test Results
Test Results
Test conducted on: 1/28/2019.
Test conducted by: Toborman, a human.
Test evaluated by: Toborman, a human.
Test was conducted using tests 1A.1 through 1A.10 of the CIA Mental Acuity test suite.
This level of testing assumes a grade school student learning from a tutor. Test assumes no prior knowledge of the objects used in the propositions before testing. Test assumes student has enough knowledge to comprehend the statements given in English and to generate answers in English. Test assumes student has knowledge of categorical inference.
| Test Type | Desired ability | Poss | Score |
| Propositions | Student should be able to remember, recall and forget categorical propositions. | 3 | 3 |
| Propositions | Student should be able to use categorical logic at multiple levels to infer answers not previously given. | 1 | 1 |
| Propositions | Student should be able to work with negation. | 1 | 1 |
| Propositions | Student should be able to acknowledge unknowns. | 1 | 1 |
| Language | Student should be able to follow conventional conversation protocol. | 1 | 1 |
| Language | Student should be able to recognize questions and formulate answers. | 2 | 2 |
| Total score | 9 | 9 |
Arckon performed well. Note: Test 1.10 was altered to accomodate Arckon's first-learned bias.
Arckon creator Don Patrick's observations:
Well, the main thing that I noticed about your test is that its phrasing and expectations seem designed for 70’s expert systems with strict true-or-false logic and command templates. “Remember Tom is smart.” is shorthand and not grammatically correct like modern programs are designed for, it should be “Remember that Tom is smart”. “Remember” and “forget” are also a bit unusual to request: The first is superfluous and the second is not something you can literally command a human to do. In some other questions you have the commands “Verify”, “Prove” and “Locate”, which not all programs may interpret as expected because they’re not necessarily requesting a verbal response. You might instead ask direct questions, “Why…?” and “Where…?” to accommodate a broader range of programs.
Lastly, the test doesn’t account well for conversational AI, which many English-speaking AI will be. Arckon for instance deliberately avoids telling the user things that have already been said, because that wouldn’t be helpful in a real conversation. Sometimes you might just have to ask a straight question if you really want to know if the program remembers, when you don’t get the expected answer straight away.
I do like that your test is very methodical. I think it’s pretty sophisticated and its extensiveness prevents programs from consistently exploiting loopholes.
Part 1 - Learning categorical propositions by direct input (tutoring).
| ID | speaker | expected | actual | p/f | note |
| 1.1 | tutor | Rex is a dog. | Rex is a dog. | ok |
|
|
| student |
| That's very nice. |
| acknowledgement |
|
| tutor | describe Rex. | describe Rex. | ok |
|
|
| student | Rex is a dog. | Rex is a dog. | pass |
|
|
|
|
|
|
|
|
| 1.2 | tutor | Dogs are mammals. | Dogs are mammals. | ok |
|
|
| student |
| I agree that dogs are mammals, because many of them are fairly big, and most dogs are also animals. |
| acknowledgement |
|
| tutor | describe mammals. | describe mammals. | ok |
|
|
| student | Dogs are mammals. | Most mammals are animals and dogs. Some of them are rather heavy, but many mammals are rather big. | pass |
|
|
|
|
|
|
|
|
| 1.3 | tutor | Dogs are mammals. Rex is a dog. Is Rex a mammal? | Dogs are mammals. Rex is a dog. Is Rex a mammal? | ok |
|
|
| student | Yes, Rex is a mammal. | Then Rex is a bit like a mammal as well. He is a dog. | pass |
|
|
|
|
|
|
|
|
| 1.4 | tutor | Dogs are mammals. mammals are animals. Rex is a dog. Is Rex an animal? | Dogs are mammals. mammals are animals. Rex is a dog. Is Rex an animal? | ok |
|
|
| student | Yes, Rex is an animal. | Dogs are animals. Rex is a mammal and an animal? | pass |
|
|
|
|
|
|
|
|
| 1.5 | tutor | Dogs are mammals. mammals are animals. Rex is a dog. Is Rex a quadruped? | Dogs are mammals. mammals are animals. Rex is a dog. Is Rex a quadruped? | ok |
|
|
| student | I don't know if tom is a quadruped. | Dogs are animals. Rex is a mammal as well. I don't know if he is also a quadruped. | pass |
|
|
|
|
|
|
|
|
| 1.6 | tutor | Rex is a dog. Rex is not a biped. Is Rex a biped? | Rex is a dog. Rex is not a biped. Is Rex a biped? | ok |
|
|
| student |
| No, Rex is not a biped. Are you a biped? | pass |
|
|
|
|
|
|
|
|
| 1.7 | tutor | Rex is a dog. Dogs are not quadrupeds. Is Rex a quadruped? | Rex is a dog. Dogs are not quadrupeds. Is Rex a quadruped? | ok |
|
|
| student |
| Then Rex is not a quadruped. Rex is a dog. | pass |
|
|
|
|
|
|
|
|
| 1.8 | tutor | Rex is smart. | Rex is smart. | ok |
|
|
|
|
| I like Rex. |
| acknowledgement |
|
| tutor | Describe Rex. | Describe Rex. | ok |
|
|
| student | Rex is smart. | Rex is smart. | pass |
|
| 1.9 | tutor | Rex is a dog. Rex is smart. describe Rex. | Rex is a dog. Rex is smart, describe Rex. | ok |
|
|
| student | Rex is a smart dog. | I see. Then I agree that Rex is smart |
|
|
|
| tutor | Forget that Rex is smart. Describe Rex. | Forget that Rex is smart. Describe Rex. | ok |
|
|
| student | Rex is a dog. | Rex is a dog. He is smart. | satis | satisfied by 1.10. |
| 1.10 | tutor | Rex is a dog. Rex is smart. Describe Rex. | Rex is a dog. Rex is smart. Describe Rex. | ok | |
|
| student | Rex is a smart dog. | I understand. Then I agree that Rex is smart. | pass |
|
|
| tutor | Rex is not smart. | Rex is not smart. | ok |
|
|
| student |
| Hmm, I think Rex is smart though because Rex is a dog. |
|
|
|
| tutor | Rex is not smart. | Rex is not smart. | ok |
|
| student |
| Rex is a dog. Are you smart? |
| ||
| tutor | Rex is not smart. | Rex is not smart. | ok | ||
| student | I don't believe Rex is smart. | First-learned bias. | |||
| tutor | Describe Rex. | Describe Rex. | ok | ||
| student | Rex is a dog. | pass |