Query Schema Automation
smolinari opened this issue · 24 comments
Hey Kristijan,
Sorry if this is the wrong place, but this project comes the closest to what I would consider getting to a possible GraphQL schema creation automation system. I mean, theoretically the models that are created are close to what would also be in a GraphQL schema, right? Can you picture automating GraphQL schema at some point, instead of doubling up on type/ model creation?
Scott
Theoretically yes, but I believe you'll be pretty limited and locked regarding how the models should be used. I like the idea though. Maybe as related module (e.g. graphql-rawmodel)?
Let me explain a bit further ... So let's take the User model for an example. I usually use multiple User{something} models in mutations because different mutations have different field validations/handlers.
const SignUpUser = require("../models/SignUpUser");
module.exports = async function(root, args, ctx) {
const user = new User(args);
try {
await user.validate();
await ctx.mongo.collection("users").insert(user); // _id automatically assigned to user if _id field exists on the model
} catch (e) {
await user.handle(e);
}
return user;
};In Rails for example we had a concept of forms where you wrote some sort of model per form. Here I found having multiple models (where one can extend the muster model) even easier and more clean.
I was just thinking, the models in RawModel have fields, they are typed, so that is the start to building the schema. If the model system is also built similar to what GraphQL requires from a gateway perspective, then automating the schema creation should be fairly easy. I am almost positive, if the model system is built like a graph, i.e. relationships are edges and models are nodes, then recreating it for GraphQL is the start to the automation.
I have a feeling Facebook isn't completely spilling the beans on what they are doing behind the scenes with GraphQL. My inquiry here was started because I've heard twice in presentations from Facebook team members that their 1000s of schema types and resolvers are built automatically. I am also involved in two discussions with the Apollo GraphQL team on adding metadata for authorization and schema introspection access into the schema. I am totally against making GraphQL a "God API", because authorization and even schema introspection access are application state (albeit slowly changing state) derived from business decisions and application state doesn't belong in code (which GraphQL schema is, in effect).
Scott
Hum... I agree but I'm not sure automation will work in all cases. I've tried having GraphQL schema types identical to RawModel models but it didn't always look desired. You can loose coding freedom in some edge cases (as I remember).
But still, RawModel schema automation would definitely work for usual/simple projects (already noted in the Readme) and it should be relatively simple to do. It would be nice to have some insides from Facebook (how they do it).
This guy hits the mental model nail on the head (the first 9 or so minutes).
https://www.youtube.com/watch?v=zWhVAN4Tg6M
So, basically, instead of modeling any kind of special models, any objects in the application are completely defined, no matter what the object is.
I'd even go further and say, the application graph model itself isn't in the code, but rather stored as metadata. So, the objects, field definitions and relationships are purely metadata and from that metadata any object or schema can be built. Remember, business logic shouldn't be in the code and what data models a business uses is business specific and can change according to business needs. I know programmers love their jobs, but when models are built into the code, it is a waste of time. Give business the tools to build their own models. Automation!
What I am suggesting is basically one level of abstraction higher than what RawModel offers currently. This also makes it sooo much simpler to modify the models too, because we add or remove metadata and not code.
Sure, you have the issue of recalling the app graph metadata (data model) from where it is persisted, but really, once it is in memory, that shouldn't be too much of an issue either.
From there, the rest (like authorization and validation) is also fairly easy to work with, because it is all metadata too.
Am I making any sense? LOL!
Scott
I guess mew should meet :). I think I understand your proposal but I'm not sure what would be the right implementation design here. Can you build a simple example of your idea so we can go from there?
Give me until end of July or so. I'm currently working hard to support another project (and programming is my hobby too). I'm really glad you are interested in my ideas. I have to be honest though, they aren't really new either. The ideas sort of come from Saleforce.com's platform. However, their perspective on their platform and how it's been developed suck IMHO. LOL! Anyway. I'll get back to you for sure. Thanks for the interest! 👍
Scott
Hehe, great! :)
This is nagging me and while I was trying to picture my idea some more, I figured, my idea must be called metadata driven programming, so I did a search for those exact terms and low and behold. Salesforce also explains it (albeit very roughly from the user perspective).
I realize this is way more ideal for an application platform (which is what I want in the end) and not necessarily ideal for a singular business application, but the real win is that, with metadata driven programming, the business decisions, whether they be model changes, business logic changes, authorization changes, validation changes, are changes in metadata and not in the code. This means, with tooling for metadata mutation, the business can make the changes, without the need of a developer. The only time developers would be needed, is for the deeper business logic needs, like for certain workflows or complicated data manipulations/ calculations. And even then, with a standardized workflow system, that could be put into metadata too. ( I know, because Salesforce does this!).
At first glance, you might think this is limiting for the developer, and in fact, it is. But, when it is done right, it is very liberating for the business in general, because not every change is an IT/ development burdon. In a small company, where the dev or devs sit across the room, this is overkill for sure. But, at some point that company might get big, or the devs might not be in the room, so changing for the bigger business gets slower and slower. That is the main advantage, the beauty, of metadata driven programming. It's extremely agile for the business, because they have a lot more control to make changes in their own hands. And naturally, nothing should break, except the business processes themselves. And that is one of the challenges!
The other challenge and what brought me here is, once the metadata is formed and persisted for the application's domain model, to then generate the GraphQL schema for the system automatically. I am super convinced Facebook does this, and they aren't telling us about that magic either. I'm very, very certain. And if they aren't doing it, they should be. Because, imagine all the changes in code they must make over time, which are business decisions. Probably 80% at least. If all those changes are dev burdens, when they really shouldn't be, then that is a ton of money wasted.
Scott
I'm very interested in this but do you have a simple code example? I find things difficult when translating theory to the actual concept/code.
Unfortunately, I don't. I'll get working on this in July.
Think about this, until I do. Instead of defining the business logic in code, like RawModel or some ORMs or Mongoose do, we can pull out metadata from a database to form any models, validation rules, authorization rules, workflow, etc. The metadata forms the business application graph. With this metadata, the system is only responsible for building...hehe.....itself. The API, the business logic objects and/ or code and even connections to databases, are created once (or on the fly) at runtime. All that is also needed is a backend UI (or a CLI for the hardcore devs LOL!), which allows for manipulation of this metadata. It allows for graphical visualization of workflow. It allows non-developers to interact with business logic directly. Yes, it "depowers" the developer some. But, it also puts responsibilities in the hands of the right people too. That is enormously valuable. 😄
Again, give me some time. I'm not the greatest or fastest programmer, being it is my hobby, and I am working on another important project for my "dream" currently.
Thanks again for the interest.
Scott
I've done a bit more research and every article on creating a DSL says to first develop the code that the language needs to compile to first (which, when you think about it, is a no-brainer actually). So, I'll be working on the logic first, where RawModel is a supporting solution for sure. I hope you won't mind, if I come back at you later with more questions? 😄
Scott
Not at all. I'm happy you started this discussion ...
Hey, so I found this suggestion really interesting. I am currently working on a project which could really use something like this! I have written some code to transform RawModel js classes into GraphQL Types. Even with nested types. It is not online yet, because I am still testing.
Also I am looking for a way to derive mutations and queries from the model, but as GraphQL need types for the arguments and return value and I am not using TypeScript, I am kind of stuck... so with this I think it is interesting to think about a strong typing system for functions.
I dont know if this is the goal with this project but I guess it could work easily by modifying the fields??
Any thoughts?
Greetings, i7clock
Thanks @i7clock for your thoughts.
The main purpose of RawModel is having an upopinionated, GraphQL type-like models that you can use for different things (backend model for saving data, backend and frontend data validation, forms ...). Instead of having a heavy all-in-one ORM model, RawModel provides an advanced general-purpose model which does not depend on any database. The idea is to pass a context object into the model (GraphQL style) or model methods to get ORM behavior. RawModel is also a good fit for building GraphQL resolvers. You can read more about this here.
RawModel represent a good fundamental building block for other functionalities but we can extend functionalities further. GraphQL schema automation could be one of these new features. I don't know exactly how to achieve that by not limiting a developer so let's continue the discussion here and maybe we somehow resolve that. If you have any concrete proposals, please post it here.
I don't know exactly how to achieve that by not limiting a developer so let's continue the discussion here and maybe we somehow resolve that.
Actually, that is a really good point, and to be honest, that is exactly my intentions. You see, the "model" (as in objects and fields, workflow, validation rules, etc.) shouldn't be a developer's domain. If she does have to program them, then she always has to ask the business what it is they need. So, all the "domain" stuff should be the in business' hands to do with as they please and with point and click functionality.
Now you are probably asking, then what is the developer supposed to do?
And the answer to that question is easy. Building out interfaces. It doesn't matter which ones, any and all of them. 😄 i.e. front-end interfaces (to humans) using the objects and fields that business created. Or APIs, so data can flow between systems. That's where the "business" domain stops and IT starts.
I realize this concept isn't the norm (currently), but systems like Salesforce show that it can be done. And Salesforce is based on metadata. Salesforce is also poised to become the 4th largest software vendor in the world too.
At any rate. I'd love to see what you've got, when it's ready, Konstantine ( @i7clock ). Your work sounds very interesting and thanks for joining in here and sharing your thoughts and work.
Scott
Thanks Scott! I am glad I found RawModel and this conversation! Also @xpepermint Thanks for going into detail on the vision of RawModel! I am sure that the concept of RawModel being a general purpose model for building ORM behaviour is something to pursue!
In my project I have to handle multiple databases (Cassandra and MongoDB) and the API is supposed to be GraphQL which is a real challenge, because Cassandra is only used to write data and the MongoDB (mongoose) database is used to read data. This means queries and mutations can be strictly separated. Additionally I keep the current states of the objects in-memory. That means I basically have to declare the data structures 4 times... (ok a little less than 4 times because of differences between the writing and reading model) so the consequence is to declare once and derive by behaviour and type.
I don't know if this is the right way, but I think modelling data multiple times is just a waste of time, especially with changing APIs and changing data models.
Thanks for your replies!
Greetings, Konstantin
@i7clock Your challenge is easily solvable by creating a context object and then passing it down to your GraphQL schema.
Quick example:
module.exports = class Context {
constructor(env) {
this.env = env;
this.cassandra = null;
this.mongo = null;
}
async connect() {
this.mongo = await MongoClient.connect(this.env.mongoUrl);
this.cassandra = await ...
}
async close() {
await this.mongo.close();
this.mongo = null;
await this.cassandra.close();
this.cassandra = null;
}
}const ctx = new Context(...);
await ctx.connect();
...
let res = await graphql(schema, '{ hello }', root, ctx);// somewhere in your app
const model = new MyModel();
await model.myMethod(ctx);
await MyModel.myMethod(ctx);I think modelling data multiple times is just a waste of time.
Not to mention, the definitions of the data model come from the business, not from the programmer. Which means, getting the data model and understanding it, is also a chore.
The thing is, the ability to do data modeling shouldn't even be "hard coded" or rather "a programmer's chore". There should be a way to generate the code needed to work with data models/ domain objects/ workflow definitions/ validation rules/ etc. When business creates their objects, the system should translate it into the necessary code in the background (which is, btw, an interface! 😄 ). It's metadata programming.
I hope you all are on the same wavelength as I am. I'm going to be honest too, you both probably have a ton more experience as programmers than I do. I, on the other hand, have many years of experience with online software (SaaS). You could call me a super power user/ administrator. And I know, metadata systems are going to be the future, because they put the line between programmers coding and data modeling in the right place. When done right, it takes away the chore work for the developer of creating data models and puts it in the hands of business, where it belongs. 😄
Scott
@xpepermint Thanks for the example!
I wouldnt consider myself an experienced programmer, as I only started 5 years ago. I havent done enterprise software projects and I come from OO C like languages, which means I am only scratching on the surface of the possibilities of JavaScript at the moment.
I am very interested in learning more about metadata programming!
One question though: What exactly do you mean by 'business'?
And in that context: Are you talking about systems only used internally of a company?
Greetings Konstantin
What I mean by "business" is, it's the people using the software and/or the people making the decisions for the people using the software and/ or the people administering the software. It could be only people in the business. It could be customers of the business. It could only just be plain users. It could be a mix of some or all of them. In the end, it's the users of the software, who add value to it. Does that make any sense? 😄
As an example. Let's say you want to create a "Spell Book" application for storing spells from certain games. (Yes, this is a real example, which I just ran into today! 😄 ). You are a programmer. Maybe you are also building this app for your own purposes. But then, you must also have a need for a Spell Book and the "business knowledge" to build it. So, you are both the business and the programmer.
Another example. You are a programmer in a corporation. The "business" needs a "Corrective Action Request" system (also a real example I personally worked on). This CAR system is a quality feedback loop for defects of the business' products in the field. It's a process for reporting and fixing quality defects. The business, in this case, is also the people who will be using this process and their managers. You, as the programmer, would have to ask the business about what data is needed, what forms are needed to enter data, what workflow is required, etc. etc. They are the ones adding the value to the (then automated) process. You are only a translator of their needs to code (and in 80% of the time spent, it is a waste of your time!).
If you are a friend of OOP and know about Bob Martin, he also defines the "business" as "the reasons for change" and I'd add to that, "which add value to the systems/ processes". 😄 Bob uses this definition in terms of explaining the Single Responsibility Principle, but I think it is generally valid, albeit a bit altered to fit my needs.
Oh, and JavaScript is a great language that gets a lot of flack, because it isn't like classical OOP. If you think of JavaScript like an abstraction above classical OOP though, where classes aren't needed and you only work with objects and inherit through object prototype delegation, then you'll find JavaScript to be heavenly. Always remember! There are no classes in JavaScript! (despite the class syntax in ES6) 😄
If that confuses you at all. Please have a look at Frontend Masters and Kyle Simpson's course on Advanced JavaScript (scroll down to see the course contents). It's actually an advanced look at the basics of JavaScript and it was a major league eye opener for me.
Scott
Thanks for clarifying what you meant by business! 😄
I understand the concepts of JavaScript, but for me it is still quite repelling to be able to modify everything at any time. I just need to get used to it I guess 😃
Yeah, but who is all that protection in classical OOP languages really protecting? If someone decides to mess with the internal workings of a "developed" object in JavaScript, it's clearly at their own risk (and ignorance).
Or put another way, good programmers won't mess with the internals of "constructed" objects or rather, they know they shouldn't. They know they have APIs to work with and should stick to them, because that is the only way they are guaranteed things will work properly (or can be properly troubleshot, when they don't work). It's the "contract" between programmers. Does the language really need to enforce those contracts, if it's understood by all devs that everyone should work with the APIs only? I don't think so.
So in the end, all that "protection" or "encapsulation" is a given, and writing extra code for it is just a waste of time (sort of like your complaint of having to deal with data modelling.) 😄
And just because you know the possibility to mess with the internals of objects is there, it doesn't mean it's inherently bad (and certainly shouldn't be repelling). It's just a difference to what you are used to and knowing it saves you time in the end, should be a form of freedom. 😉
Ok enough of my JavaScript fanboyism. 😄 Back on topic.
Scott
Feel free to reopen this issue.