Introduction

The following blog posts have been written and revised over the past few years. The topics addressed often fall into more than one category, but I have organized them into sections that reflect what I think may be a reader's primary area of interest.

My posts reflect a few of my interests and experiences in developing specific applications and do not attempt to provide either complete surveys or plumb the full depth of the topic areas being addressed. I hope that they intrigue you so you can share my enthusiasm for what I have worked on in the past and for the challenges yet to come.

One of the directors of Arity years ago told me that no one should go into business to make money. They should go into business to do something worthwhile. If they do it well (and perhaps have a bit of good luck) then they may make money. By extension, no one should write a blog merely as a vanity exercise. They should write something that is of worth to their potential audience. I have tried to write some articles that provide some information and some perspective that is not readily available on the first page of a Google search. The opinions that I express are certainly not universally shared. But the facts that I present are not "alternative facts".

Some of the articles here are still being written or edited. In some cases I have not included the article pending completion. As with most people's blogs I expect to be adding more content over time.

This blog has been written using Documenter. This provides a dialect of Markdown and LaTeX with extensions that are integrated into the Julia language. I have more to say about Julia and Documenter in an article in the Programming section of this blog.

Below I provide a short introduction to each section of this blog.

1. Big Data

Many people are confused by the term "Big Data". What constitutes "big" is ever changing and is relative to the problem being tackled.

"Big data" problems may be defined roughly as those that involve both very large quantities of data (typically at least a terabyte but often orders of magnitude greater) and involve processes with one or more sub-tasks that access or modify significant fractions of the total amount of data under management. In contrast, ordinary transaction processing systems (such as a DMBS supporting retail operations) may manage terabytes or petabytes of data but if each transaction only accesses or modifies a tiny fraction of the data then it does not qualify as a "big data" problem.

"Deep learning" is sometimes conflated with big data in non-technical literature. Generally speaking, they are different disciplines, although neural models may be used to provide inferential services within big data processing.

I address some of the nuts and bolts that hold Big Data processing together. The entry on an architecture for workflow addresses an unusual topic but one that has been important for a real-world application.

2. Reasoning and Decision Making under Uncertainty

Blog articles in this section were inspired by problems that are interdisciplinary in nature but touch in some way to reasoning processes.

Uncertainty may arise in a few ways. The problem may have an inherent probabilistic nature such as when events are generated according to some known or unknown distribution. Or, some of the data that contributes to a decision may be missing. Or, the problem may itself be undecidable, meaning that it is a logical impossibility to construct an algorithm that is guaranteed to complete with a correct answer.

Automated reasoning and decision-making algorithms are commonly regarded as part of AI, particularly when those problems that require learning, and most particularly when the form of instruction has a vague or indirect relationship with the ability to learn the association between stimulus and action.

My blog entries have so far not directly addressed two areas that I am most interested in: causal probabilistic reasoning and reinforcement learning. These enjoy an abundance of accessible resources. I've chosen to write about some topics which I find to be fun, under-reported and thought-provoking.

3. AI, NLP and KR&R

My blog articles in this section are primarily about symbolic approaches to Artificial Intelligence. This is not because I think less of the astonishing successes of neural approaches and the truly remarkable progress of the last few years. Rather, it is because the factors that gave rise to the most recent advances in neural approaches have analogies that will in the new future propel symbolic approaches. In addition, new synergies between neural and symbolic approaches are emerging to tackle ever deeper problems.

Natural Language Processing (NLP) has roots in grammatical analysis going back to the 5th or 6th century BCE in ancient India. The first known treatise analyzing Sanskrit semantics and syntax was written by Panini in the 4th century BCE. The methods and modes of analysis have changed but the central claim has persisted: That an understanding of the structure of language is nothing less than an understanding of the nature of thought and intelligence, although as yet this remains unfulfilled. This view is in stark contrast to superficial processing of text or utterances provided by statistical text analytics or neural processing.

Likewise, the formalization of logical reasoning methods stems from traditions equally ancient. Most of the underlying basis for ontological reasoning and the principles for organizing knowledge were known to the greats of Greek philosophy.

Many forms of Knowledge Representation and Reasoning (KR & R) have been developed and applied to different problems. The span of ideas and application areas is astonishing but remains Balkanized. Lines of research take different avenues often with very different starts such as logic, probability, metaphors for space and time, networks of links and nodes, and models of "common sense".

Surprisingly, the gulf between models of language semantics and KR & R remain wide except for very simple applications. One reason became very evident with the development of natural language query interfaces to databases starting in the 1970s. The imprecision of language created an "impedance mismatch" with the formality required for DBMSs. ("Give me all customers who own a dog and a cat." vs. "Give me all customers who live in NY and CT." The word "and" seems to indicate a disjunction in the first case and a conjunction in the second case.) Other examples stem from propositional attachment ambiguity, negation in various forms, anaphora, ellipsis, and other common linguistic phenomena.

The creation of a "general artificial intelligence" that extends beyond the very specialized applications that have been developed to date will require developments in NLP, KR & R and learning and will most certainly combine neural and symbolic methods.

My articles in this section related to problems that I wrestle with on a regular basis. Some are merely sketches of an idea. Others span a particular challenge that has broad applicability.

4. Logic Programming and Prolog/64

Almost 30 years have passed since I designed Arity/Prolog32 (and I started learning about Prolog in 1980 and started implementing my first Prolog in 1983). Arity/Prolog32 has stood up incredibly well and still is the fastest Prolog, beating even the 64 bit version of SICStus Prolog. On the other hand, it shows its age in several aspects such as the object format generated, development environment, and of course, that a 64 bit version would allow tight coupling into other 64 bit code.

Blog articles in this section are a beginning of a more complete discussion about the future of logic programming and Prolog (being best to not labor over the past).

5. Applications as if Cognitive Science Matters

People are swept up in the hype about Artificial Intelligence. 'nuff said.

I wish people were more enamored with Cognitive Science and, in particular what it has to say about the user experience and user interface of applications.

The articles in this section only touch on applied cognitive science but perhaps a good start.

6. Numerical Methods

While I have spent much of my time immersed in symbolic methods, I love diving into numerical problems.

In grad school at MIT I took a course that was ostensibly about statistical modeling and diagnostics but even more was a deep dive into numerical analysis. On the first day of class there were about 45 students. The Professor Roy Welsch looked out at the group and said that he expected that only 15 students would complete the course and that he would award an A to to the top five, a B to the next five and a C (which is a failure in grad school) to the rest. I decided to stick with it even though most of the students had a much stronger background for this course than I had. Well, it turned out that indeed, about 15 students completed the course and when final rankings were published I was number 5. He awarded me a B+ and I was pissed. But I also had worked harder in that course and learned more in that course than any other single course that I had ever taken. So I could not be too angry.

The articles in this section are in no small part a way of confirming that I understand some knotty problems by the best way possible - explaining to others, in this case, to you, dear reader.

7. Programming

I never have considered myself a programmer (let alone God Forbid! a "coder"). Writing programs is something I do to express ideas and occasionally make some money.

My father taught me BASIC when I was in grade school using a teletype that communicated by dial-up with a timesharing service. Connectivity was so expensive that programs were first entered offline on paper tape.

In high school I started writing compilers and interpreters, including an interpreter for BASIC in APL. To this day I am still writing compilers and interpreters.

I have programmed in many languages. My favorites are quite different in character: Prolog, JavaScript, assembly language, C, and more recently Julia. I harbor great antipathy toward classical object-oriented languages such as C++ and Java. Get me started and I will tell you how these languages have retarded progress in software practice and ruined otherwise smart people's ability to tackle larger problems.

Articles about Prolog have been put in their own section. Here I provide some advice about JavaScript and some observations about Julia.

8. Recommended References

I love to recommend books and articles to others. Generally, I cannot help but give a little commentary along the way. Hence this section and its content.

9. Miscellaneous Posts

I've placed a few articles in this section that are apropos of none of the sections above.