node.js Deeplab AI image classification not working on node.js tensorflow

[TheNuzziNuzz]

Tensorflow model deeplab: https://github.com/tensorflow/tfjs-models/tree/master/deeplab doesn't seem to work for me on node.js.

I haven't been able to find any example code for deeplab in nodejs. When I try to segment an image I get the error

error: The dtype of dict['ImageTensor'] provided in model.execute(dict) must be int32, but was float32

And googling the error seemed to turn up no helpful soloutions.

 

Would love thoughts from sombody who knows more!

[TheNuzziNuzz]

44 minutes ago, zhnu said:

Easy you're passing a float instead of an int. Also maybe consider using tensorflow lite with the pre-trained models https://www.tensorflow.org/lite/models/image_classification/overview

They don't seem to have a nodejs api for lite. What is the benefit of using lite over regular?

[TheNuzziNuzz]

2 minutes ago, zhnu said:

Lighter to process. On that page you have a pre-trained model should work for what you're doing even if you're using regular tf.

I think it just uses mobilenet

https://www.tensorflow.org/lite/guide/hosted_models#image_classification

[TheNuzziNuzz]

4 minutes ago, zhnu said:

https://tfhub.dev/s?deployment-format=tfjs&module-type=image-classification&tf-version=tf2

Neat thank you! How do I import these models into node? Do I just point tf.node.loadSavedModel to the tar.gz?

[TheNuzziNuzz]

6 minutes ago, zhnu said:

tf.loadGraphModel("https://tfhub.dev/google/tfjs-model/imagenet/inception_v1/classification/3/default/1", { fromTFHub: true })

 

How do I actually run the model? There is no documentation on the usage page. Thank you so much for helping me, I know this is such a noob situation, I'm not sure why i'm having such a hard time finding any tensorflow node.js documentation. I really appreciate your time.

[TheNuzziNuzz]

const predictions = await model.predict(tf.node.decodeImage(frameData))

results in

Error: The shape of dict['images'] provided in model.execute(dict) must be [-1,-1,-1,3], but was [480,720,3]

 

[TheNuzziNuzz]

12 minutes ago, zhnu said:

I think you really need to start with a tutorial before jumping right in:
https://developer.ibm.com/technologies/artificial-intelligence/tutorials/an-introduction-to-ai-in-nodejs/

I have gone through this, I understand it for the model in the example. However i'm not sure how that translates when using the models on TF hub. TF hub doesn't provide a link to the labels like is used in the example you cited.

[TheNuzziNuzz]

1 minute ago, zhnu said:

Yes it does for example:

https://tfhub.dev/google/imagenet/inception_v1/classification/4
https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt

Okay, thank you. I'm really just a disorganized mess. I'm used to the flow of most npm packages where the node usage is described in the package page. Whereas with this the information is scattered and there is no copy/paste example code for each model.

 

So I tried running the example code above for the inception model, and I got:

TypeError: Cannot read property 'dataSync' of undefined

at this point in the example: 

 const extractClassesAndMaxScores = function (predictionScores) {
  console.log('calculating classes & max scores')

  const scores = predictionScores.dataSync()
...

 

[TheNuzziNuzz]

9 minutes ago, zhnu said:

predictionScores is going for some reason undefined to the function

        const prediction = await model.execute(tf.node.decodeImage(frameData).expandDims(0).toFloat())
        const index = prediction.as1D().argMax().dataSync()[0]
        console.log(labels[index])

Okay I got this to work, I turned https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt into an array. 

sadly the results are completely inaccurate and nonsense. Thank you for your help!

[TheNuzziNuzz]

3 minutes ago, zhnu said:

The results have a score you have to look at the score and decide the threshold, but yeah is very easy to have false positives.

Is there any such way to get locally run models to be as accurate as google's Cloud Vision?

[TheNuzziNuzz]

23 minutes ago, zhnu said:

You can try different models see what works best, but I doubt you can beat google cloud vision.

I want to use deeplab, but it keeps giving me 

(node:3128) UnhandledPromiseRejectionWarning: Error: The dtype of dict['ImageTensor'] provided in model.execute(dict) must be int32, but 
was float32

with

        const canvas = createCanvas(720, 640)
        const ctx = canvas.getContext('2d')
        ctx.drawImage((await loadImage(frameData)), 0, 0)
        model.segment(canvas)

 

[TheNuzziNuzz]

I figured it out! Deeplab only works with version 1.3.1 of tensorflow. I downgraded my packages and it is working amazing.